<?xml version="1.0" encoding="UTF-8"?><rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Clint Parker</title><description>Writings on software development, testing, and architecture.</description><link>https://clintparker.com/</link><item><title>Speckit Workflows</title><link>https://clintparker.com/blog/speckit-workflows/</link><guid isPermaLink="true">https://clintparker.com/blog/speckit-workflows/</guid><description>Spec Kit workflows can drive the whole specify, plan, tasks, implement sequence, so I wrote a gate-free YOLO workflow and published it as an add-on.</description><pubDate>Tue, 28 Jul 2026 23:23:13 GMT</pubDate><content:encoded>&lt;p&gt;Is your team using &lt;a href=&quot;https://github.github.io/spec-kit/&quot;&gt;Spec Kit&lt;/a&gt;, but you hate having to remember the sequence &lt;code&gt;/speckit-specify =&amp;gt; /speckit-plan =&amp;gt; /speckit-tasks =&amp;gt; /speckit-implement&lt;/code&gt; ? you know it&apos;s going to get it right, and let&apos;s be honest, you&apos;re not going to review all of that intermediate output anyway (you should, though).&lt;/p&gt;
&lt;p&gt;Well, you should check out &lt;a href=&quot;https://github.github.io/spec-kit/reference/workflows.html&quot;&gt;Spec Kit workflows&lt;/a&gt;. I don&apos;t see much discussion of this feature in the Spec Kit community. I stumbled across it while going through a new project the other day, and it took me a while to figure out how to even invoke it (which is a little embarrassing). But once you find it, it&apos;s a pretty straightforward command you run from the terminal, and it starts up Claude itself and drives the whole thing.&lt;/p&gt;
&lt;p&gt;I ran the workflow that ships with the standard Spec Kit install, and it had the same basic gates as running the commands by hand. On one hand, that&apos;s nice — you don&apos;t have to remember the sequence anymore. On the other hand, sometimes I just want it to go all the way to the end. This is why we have source control. This is why we have branches. If it goes sideways, I throw away the branch and try again.&lt;/p&gt;
&lt;h3&gt;YOLO mode&lt;/h3&gt;
&lt;p&gt;So I wrote my own version of the shipped workflow, called YOLO. It removes the gates and runs through to completion. Invoking it is as simple as:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;specify workflow run yolo -i spec=&quot;make the app do the thing&quot;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Here&apos;s the workflow definition (.specify/workflows/yolo/workflow.yml):&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;schema_version: &quot;1.0&quot;
workflow:
  id: &quot;yolo&quot;
  name: &quot;Full SDD Cycle - no gates&quot;
  version: &quot;0.0.1&quot;
  author: &quot;clintcparker&quot;
  description: &quot;Runs specify → plan → tasks → implement without review gates&quot;

requires:
  # 0.8.5 is the first release with engine-side resolution of the
  # ``integration: &quot;auto&quot;`` default. Older versions would treat &quot;auto&quot;
  # as a literal integration key and fail at dispatch.
  speckit_version: &quot;&amp;gt;=0.8.5&quot;
  integrations:
    # The four commands below (specify, plan, tasks, implement) are core
    # spec-kit commands provided by every integration. The list here is an
    # advisory, non-exhaustive compatibility hint following the documented
    # ``any: [...]`` schema -- it is NOT a closed set. The workflow runs
    # against any integration the project was initialized with, including
    # ones not listed below, as long as that integration provides the four
    # core commands referenced in ``steps``.
    any:
      - &quot;claude&quot;

inputs:
  spec:
    type: string
    required: true
    prompt: &quot;Describe what you want to build&quot;
  integration:
    type: string
    default: &quot;auto&quot;
    prompt: &quot;Integration to use (e.g. claude, copilot, gemini; &apos;auto&apos; uses the project&apos;s initialized integration)&quot;
  scope:
    type: string
    default: &quot;full&quot;
    enum: [&quot;full&quot;, &quot;backend-only&quot;, &quot;frontend-only&quot;]

steps:
  - id: specify
    command: speckit.specify
    integration: &quot;{{ inputs.integration }}&quot;
    input:
      args: &quot;{{ inputs.spec }}&quot;

  - id: plan
    command: speckit.plan
    integration: &quot;{{ inputs.integration }}&quot;
    input:
      args: &quot;{{ inputs.spec }}&quot;

  - id: tasks
    command: speckit.tasks
    integration: &quot;{{ inputs.integration }}&quot;
    input:
      args: &quot;{{ inputs.spec }}&quot;

  - id: implement
    command: speckit.implement
    integration: &quot;{{ inputs.integration }}&quot;
    input:
      args: &quot;{{ inputs.spec }}&quot;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Just drop this file into the &lt;code&gt;workflows&lt;/code&gt; directory inside your &lt;code&gt;.specify&lt;/code&gt; directory. Then make sure you register it in the Spec Kit workflow registry (.specify/workflows/workflow-registry.json):&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;    &quot;yolo&quot;: {
      &quot;name&quot;: &quot;Full SDD Cycle - no gates&quot;,
      &quot;version&quot;: &quot;0.0.1&quot;,
      &quot;description&quot;: &quot;Runs specify \u2192 plan \u2192 tasks \u2192 implement without review gates&quot;,
      &quot;source&quot;: &quot;local&quot;,
      &quot;installed_at&quot;: &quot;2026-07-28T21:16:32.525333+00:00&quot;,
      &quot;updated_at&quot;: &quot;2026-07-28T21:16:32.525333+00:00&quot;
    }
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;That&apos;s it. No gates, no babysitting 😊 kick it off and come back to a finished branch.&lt;/p&gt;
&lt;h3&gt;It&apos;s better than I expected&lt;/h3&gt;
&lt;p&gt;Here&apos;s the part that made me stop what I was doing. I kicked off a run on a project that has pretty rigid behavioral tests and behavioral testing guidelines set up, and this particular change touched a UI component. During the implementation phase, during the &quot;implement&quot; phase, Claude spawned Playwright on its own. I was doing other stuff on the machine, and I saw the browser come up and start clicking through the app, doing exactly the kind of manual validation I would have wanted to see anyway.&lt;/p&gt;
&lt;p&gt;I&apos;m running this with Claude Opus 5, and I&apos;m extremely impressed. It comes back to the same thing it always comes back to: having a better idea of what you want to build at the end of the day. But this one knocked it out of the park. It was exactly like telling a competent engineer on my team, &quot;hey, make this change&quot; ... and they did everything right. They followed every step, they documented it, they tested it.&lt;/p&gt;
&lt;p&gt;Anyway, happy coding.&lt;/p&gt;
&lt;h3&gt;Update (2026-07-29): you don&apos;t have to copy-paste this anymore&lt;/h3&gt;
&lt;p&gt;I&apos;ve published YOLO as a proper Spec Kit add-on: &lt;a href=&quot;https://github.com/clintcparker/speckit-addons&quot;&gt;clintcparker/speckit-addons&lt;/a&gt;. Skip the &quot;drop this file in &lt;code&gt;.specify/workflows/&lt;/code&gt; and hand-edit the registry JSON&quot; dance above — Spec Kit has a catalog mechanism, and now there&apos;s a catalog to point it at.&lt;/p&gt;
&lt;p&gt;Register the catalog once, then install:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;specify workflow catalog add https://raw.githubusercontent.com/clintcparker/speckit-addons/main/workflows/catalog.json
specify workflow add yolo
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Or grab just this one workflow without registering anything:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;specify workflow add yolo --from https://raw.githubusercontent.com/clintcparker/speckit-addons/yolo-v0.1.0/workflows/yolo/workflow.yml
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Either way, &lt;code&gt;specify workflow run yolo -i spec=&quot;make the app do the thing&quot;&lt;/code&gt; still works exactly the same.&lt;/p&gt;
</content:encoded></item><item><title>Git Ignore ... and Untrack!</title><link>https://clintparker.com/blog/git-ignore-and-untrack/</link><guid isPermaLink="true">https://clintparker.com/blog/git-ignore-and-untrack/</guid><description>Adding a tracked file to .gitignore does nothing, so I built a VS Code extension that untracks and ignores it in one step.</description><pubDate>Thu, 25 Jun 2026 18:53:47 GMT</pubDate><content:encoded>&lt;p&gt;We&apos;ve all done it. You commit a file you shouldn&apos;t have. A &lt;code&gt;.env&lt;/code&gt;, a &lt;code&gt;node_modules/&lt;/code&gt; you forgot to ignore, some local config, a stray build folder. So you add it to &lt;code&gt;.gitignore&lt;/code&gt;... and git keeps tracking it anyway.&lt;/p&gt;
&lt;p&gt;That&apos;s because &lt;code&gt;.gitignore&lt;/code&gt; only governs &lt;em&gt;untracked&lt;/em&gt; files. Once git is already tracking something, ignoring it does nothing. 🤦‍♂️ The actual fix is two steps:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;git rm --cached path/to/file
echo &quot;/path/to/file&quot; &amp;gt;&amp;gt; .gitignore
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;I do this often enough (and yet I &lt;em&gt;always&lt;/em&gt; have to look it up) that I finally built a VS Code extension to do it for me. (Claude helped)&lt;/p&gt;
&lt;h2&gt;Ignore &amp;amp; Untrack&lt;/h2&gt;
&lt;p&gt;&lt;a href=&quot;https://github.com/clintcparker/git-ignore-and-untrack&quot;&gt;&lt;strong&gt;git-ignore-and-untrack&lt;/strong&gt;&lt;/a&gt; adds a single &lt;strong&gt;Ignore &amp;amp; Untrack&lt;/strong&gt; action to the Source Control context menu. Right-click a file under &lt;strong&gt;Changes&lt;/strong&gt;, pick it, and in one step it:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Removes the file from the git index (&lt;code&gt;git rm --cached&lt;/code&gt;), keeping your copy on disk, and&lt;/li&gt;
&lt;li&gt;Adds an anchored entry (&lt;code&gt;/path/to/file&lt;/code&gt;) to your &lt;code&gt;.gitignore&lt;/code&gt;.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Commit to finalize, and the file is gone from version control while your local copy stays exactly where it is.&lt;/p&gt;
&lt;p&gt;It&apos;s &lt;a href=&quot;https://marketplace.visualstudio.com/items?itemName=clintcparker-ext.git-ignore-and-untrack&quot;&gt;on the VS Code Marketplace&lt;/a&gt;, the source is &lt;a href=&quot;https://github.com/clintcparker/git-ignore-and-untrack&quot;&gt;on GitHub&lt;/a&gt;, and it&apos;s licensed &lt;a href=&quot;https://github.com/clintcparker/git-ignore-and-untrack/blob/main/LICENSE&quot;&gt;0BSD&lt;/a&gt; — do whatever you want with it.&lt;/p&gt;
&lt;p&gt;If you&apos;ve ever committed a secret and scrambled to un-commit it, this one&apos;s for you.&lt;/p&gt;
&lt;p&gt;**TBF, this won&apos;t fix your history. Rotate those keys!&lt;/p&gt;
</content:encoded></item><item><title>Software Teams Are the Worst Managed Teams on the Planet</title><link>https://clintparker.com/blog/software-teams-are-the-worst-managed-teams-on-the-planet/</link><guid isPermaLink="true">https://clintparker.com/blog/software-teams-are-the-worst-managed-teams-on-the-planet/</guid><description>The promotion trap, absent feedback, and allergy to accountability make software teams the worst managed anywhere — and the engineers underneath pay the price.</description><pubDate>Wed, 17 Jun 2026 22:46:02 GMT</pubDate><content:encoded>&lt;p&gt;I keep having this conversation. Different companies, different people, different contexts. And I keep landing in the same place.&lt;/p&gt;
&lt;p&gt;Software engineering teams are some of the worst managed teams in any industry. Maybe the worst. I&apos;ve seen it across every company I&apos;ve worked at, every company I&apos;ve talked to, and every leadership team I&apos;ve consulted with. It&apos;s pervasive.&lt;/p&gt;
&lt;p&gt;And the wild part? It&apos;s not because we don&apos;t know better. The knowledge exists. The frameworks exist. The books have been written. The concepts are not new. I&apos;ve been hearing them for over 20 years. What&apos;s new, somehow, is that experienced managers are still encountering these ideas for the first time.&lt;/p&gt;
&lt;h3&gt;The Promotion Trap&lt;/h3&gt;
&lt;p&gt;I think part of this comes down to the youth of software as an industry. We&apos;re still relatively new at this compared to other industries. Software has been able to attract the funding that and reach scales that would&apos;ve taken much more real-world success historically. Therefore, those other industries have had decades (or centuries) to codify what leadership looks like at every level. Software hasn&apos;t. We&apos;re still making it up.&lt;/p&gt;
&lt;p&gt;But the bigger issue is the promotion trap.&lt;/p&gt;
&lt;p&gt;You were good at writing code, so now you&apos;re managing people who write code. That&apos;s the logic. And it&apos;s terrible logic. These are fundamentally different jobs. Being a great engineer does not make you a great engineering manager any more than being a great pilot makes you a great airline executive. The skills overlap in some places, and an awareness of the work is critical, but the responsibilities are completely different.&lt;/p&gt;
&lt;p&gt;I&apos;ve seen this play out so many times. Someone gets promoted because they&apos;re technically strong, maybe they&apos;ve been around a while, maybe they&apos;re just the most senior person on the team. And now they&apos;ve got six or eight direct reports, and they have no idea what to do with them. They&apos;ve never been properly managed before, so why should they? They don&apos;t know how to set expectations. They don&apos;t know how to give feedback. They don&apos;t know how to run a one-on-one that&apos;s actually useful. They definitely don&apos;t know how to have a hard conversation about performance.&lt;/p&gt;
&lt;p&gt;So they don&apos;t. They just keep writing code and hope the people stuff works itself out.&lt;/p&gt;
&lt;p&gt;It doesn&apos;t. It won&apos;t. Hope is not a strategy.&lt;/p&gt;
&lt;h3&gt;How Did You Get This Job?&lt;/h3&gt;
&lt;p&gt;I go to mandatory manager training from time to time. I appreciate it. I&apos;m always grateful for it. Worst case we&apos;ll establish a common vocabulary. But I&apos;ll be honest: most of the content isn&apos;t new to me. That&apos;s fine. What shocks me is watching other managers in the room react to the material like they&apos;ve never heard it before.&lt;/p&gt;
&lt;p&gt;How did these people get into a managerial position and not understand the expectations of the role? How did the person who hired them not inspect their skill set in this space? It blows me away. And these aren&apos;t junior people. These are directors, senior managers, people who&apos;ve been in leadership for years.&lt;/p&gt;
&lt;p&gt;The concepts aren&apos;t exotic. Have regular one-on-ones. Set clear expectations. Give timely feedback. Create career development plans. Hold people accountable. These are basics. But the bar is apparently so low that &quot;basics&quot; still qualifies as a revelation.&lt;/p&gt;
&lt;p&gt;I tend to avoid hiring managers externally who haven&apos;t managed before. I had a situation where a guy was hired as a manager weeks before I started at a company. I met him, went through his background, and asked the obvious question. &quot;Wait, so this is your first time being a manager?&quot; He said yeah. I told him, straight up, I wouldn&apos;t have hired you. Not because you&apos;re not capable, but because you don&apos;t have any of the foundational skills for this particular job.&lt;/p&gt;
&lt;p&gt;I gave him two choices. Take the manager role, and I will give you a firehose of mentorship and a massive amount of unsolicited feedback. Or transition back to being a senior IC at the same comp, because you&apos;re clearly excellent at writing code. He chose the management path. And it worked out really well. But it worked because I set those expectations on day one and followed through relentlessly. Most organizations don&apos;t do that. They just hand someone a team and walk away.&lt;/p&gt;
&lt;h3&gt;The People Paying the Price&lt;/h3&gt;
&lt;p&gt;Here&apos;s the part that actually bothers me. It&apos;s not the bad managers themselves. People can learn. People can grow. What bothers me is the engineers underneath them who are being underserved and don&apos;t even know it.&lt;/p&gt;
&lt;p&gt;Think about how many engineers have never had a manager who gave them real, timely feedback. Never had a manager who built a career plan with them. Never had a manager who set clear, measurable expectations and then actually followed up. These engineers think that&apos;s normal. They think management is supposed to be absent, or reactive, or purely administrative. And they perpetuate that vision when they eventually get promoted into management themselves.&lt;/p&gt;
&lt;p&gt;It&apos;s a cycle. Bad management produces people whose only reference for management is bad management. And the engineers at the bottom of that chain just accept that this is how it works.&lt;/p&gt;
&lt;p&gt;When I see a group where somebody&apos;s got 18 direct reports, I don&apos;t think &quot;wow, that person is important.&quot; I think that&apos;s 18 people who are not getting served. Nobody can meaningfully manage 18 people. Seven is the sweet spot. Above seven, you start losing the ability to really know what each person is working on, where they&apos;re struggling, and what they need to grow. I managed 14 directs once earlier in my career, and even that was exhausting.&lt;/p&gt;
&lt;h3&gt;No Consequences, No Change&lt;/h3&gt;
&lt;p&gt;The other pattern I keep seeing is an almost allergic reaction to accountability in engineering organizations.&lt;/p&gt;
&lt;p&gt;I&apos;ve been in conversations where senior leaders have told me, with a straight face, &quot;I don&apos;t think we&apos;ll ever fire an engineer.&quot; Not because every engineer was performing well. Just because the culture was so averse to holding anyone to a standard that letting someone go was unthinkable.&lt;/p&gt;
&lt;p&gt;And it&apos;s not just firing. It&apos;s any form of real accountability. I&apos;ve talked with engineering leaders who know that certain people on their teams aren&apos;t embracing new practices, aren&apos;t meeting expectations, aren&apos;t growing. And when I ask what the consequences are for that, the answer is: there aren&apos;t any.&lt;/p&gt;
&lt;p&gt;The fear is always the same. They&apos;ve been here for years. They know where the bodies are buried. If we lose them, who&apos;s going to maintain this system? And that fear is real, but it&apos;s also a trap. It means you&apos;re holding onto underperformers because you failed to document the system, failed to cross-train, failed to build any resilience into the team. The institutional knowledge problem is a symptom of the management problem, not a justification for ignoring it.&lt;/p&gt;
&lt;h3&gt;What I Actually Do&lt;/h3&gt;
&lt;p&gt;I don&apos;t have some revolutionary framework. What I have is consistency and follow-through. Here&apos;s what I&apos;ve landed on after doing this for a long time.&lt;/p&gt;
&lt;h4&gt;Managers don&apos;t write application code.&lt;/h4&gt;
&lt;p&gt;This is probably my most controversial position. I tell every people leader in my organization: you do not contribute to the application code. But you also have to be responsible for and aware of all of the application code being written.&lt;/p&gt;
&lt;p&gt;That forces a mindset shift. Your job is not to be the best programmer on the team. Your job is to make the team successful. You participate in code reviews. You understand the architecture. You know what&apos;s being built and why. But your time goes to career development, expectation-setting, coaching, and removing blockers. If you&apos;re heads-down writing features, you&apos;re not doing your actual job.&lt;/p&gt;
&lt;p&gt;AI has made me soften slightly on this, because there are cases where a manager prototyping something with an AI tool is genuinely the fastest path. But the principle holds: your primary responsibility is the people, not the code.&lt;/p&gt;
&lt;h4&gt;Feedback is immediate.&lt;/h4&gt;
&lt;p&gt;I hate annual performance reviews. Hate them. If I&apos;m doing my job, my team members know exactly where they stand at all times. I have conversations about performance multiple times a week. Not formal sit-downs, just real talk. That went well. This didn&apos;t. Here&apos;s what I noticed in that meeting. Here&apos;s an opportunity you missed.&lt;/p&gt;
&lt;p&gt;Feedback should never be delayed. The only exception is if giving it right now would disrupt something that&apos;s actively happening. But immediately after? Definitely. I&apos;m pulling you aside. A lot of the people who still enjoy working with me will tell you that&apos;s one of the things they appreciate most. They always know where they stand.&lt;/p&gt;
&lt;p&gt;And here&apos;s the thing: when you do this consistently, the hard conversations become easy. If someone&apos;s performance is slipping and you&apos;ve been talking about it for weeks, the eventual &quot;this needs to change or we&apos;re going to have a bigger problem&quot; conversation isn&apos;t a surprise. It&apos;s a continuation. You talked about this. They knew. The expectations were there. That conversation almost writes itself.&lt;/p&gt;
&lt;p&gt;But if you saved it all up for a quarterly review? Now it&apos;s a bomb. Now they&apos;re blindsided. Now it&apos;s adversarial. You did that to them by waiting.&lt;/p&gt;
&lt;h4&gt;Be proactive about org structure.&lt;/h4&gt;
&lt;p&gt;I try to evaluate the org chart as a whole at a minimum every six months. Not because I want to restructure constantly, but because companies change. The team that made sense six months ago might not make sense today. Someone might have left. A new product priority might have shifted the workload. You have to keep asking: does this still work?&lt;/p&gt;
&lt;p&gt;I encourage all my leaders to think the same way. What if we lose someone? What criteria would we use for a reduction? Who&apos;s underperforming? What&apos;s our succession plan? These aren&apos;t paranoid exercises. They&apos;re responsible leadership. If you can&apos;t answer those questions about your team, you&apos;re not paying close enough attention.&lt;/p&gt;
&lt;h4&gt;Hire managers who have managed.&lt;/h4&gt;
&lt;p&gt;I mentioned this earlier, but I want to be clear about why. It&apos;s not that first-time managers can&apos;t succeed. The guy I told the story about earlier is proof of that. It&apos;s that the failure rate is high, and most organizations don&apos;t provide the level of support needed to make it work.&lt;/p&gt;
&lt;p&gt;If you&apos;re going to promote someone into their first management role internally, great. You have context. You know their strengths and gaps. You can build a development plan. But hiring an external candidate who has never managed before into a management role? You&apos;re gambling. And the people who lose that gamble are the engineers on their team.&lt;/p&gt;
&lt;h3&gt;The Industry Has to Grow Up&lt;/h3&gt;
&lt;p&gt;I genuinely think this is the biggest unlock most engineering organizations are ignoring. Everyone&apos;s talking about AI adoption, faster delivery, agentic development, new tooling. And those things matter. But none of them work if the management layer is incompetent.&lt;/p&gt;
&lt;p&gt;You can&apos;t build a high-performing engineering team on a foundation of absent leadership. You can&apos;t adopt new practices if managers aren&apos;t holding people accountable to old ones. You can&apos;t move fast if half the team is disengaged because nobody&apos;s ever told them what &quot;good&quot; looks like.&lt;/p&gt;
&lt;p&gt;The industry has to grow up. Software management has to stop being an afterthought. It has to stop being the thing that happens by accident when someone good at coding gets promoted. It has to be intentional, and it has to be held to the same standard we hold everything else.&lt;/p&gt;
&lt;p&gt;Because right now, the people paying the price are the engineers. And they deserve better than what most of them are getting.&lt;/p&gt;
</content:encoded></item><item><title>Strangers on the Internet, or Why PRs Are Conversations, Not Notifications</title><link>https://clintparker.com/blog/prs-are-conversations-not-notifications/</link><guid isPermaLink="true">https://clintparker.com/blog/prs-are-conversations-not-notifications/</guid><description>The open-source PR model doesn&apos;t fit internal teams; have the conversation early, stage the review, and let the PR confirm shared understanding.</description><pubDate>Sun, 29 Mar 2026 18:26:19 GMT</pubDate><content:encoded>&lt;p&gt;Here&apos;s something I say a lot that usually gets a reaction: &lt;em&gt;we are not strangers on the internet.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;The open-source PR model — push your work, notify the team, hope someone picks it up, iterate based on comments — makes total sense when contributors don&apos;t know each other and are working across time zones and organizations with no shared context. It was built for that. It works great for that.&lt;/p&gt;
&lt;p&gt;It does not work great for a team of people who sit (or video call) together every week, have shared context on the codebase, and are accountable to each other for delivery outcomes.&lt;/p&gt;
&lt;p&gt;When I watch internal teams use the open-source model, I see a few things consistently:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;PRs get big before anyone looks at them.&lt;/strong&gt; The engineer works in isolation until they think the thing is done, then opens a PR with thirty changes. The reviewer now has to reconstruct the entire intent of the work from the diff. This is expensive for the reviewer, and the engineer has already invested too much to pivot meaningfully on structural feedback.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Reviews become rubber stamps or rabbit holes.&lt;/strong&gt; Either the reviewer approves because the thing looks roughly right and they don&apos;t want to slow it down, or they find a fundamental problem and now you have a PR comment thread that should have been a twenty-minute conversation three days ago.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Coverage and quality go sideways late.&lt;/strong&gt; By the time someone notices a testing gap or an architectural issue, the engineer is context-switched out and getting back up to speed costs time neither of you wanted to spend.&lt;/p&gt;
&lt;p&gt;The model I&apos;ve had the most success with is straightforward: have the conversation early, stage the review, and use the PR to confirm shared understanding rather than initiate it.&lt;/p&gt;
&lt;p&gt;Specifically — if you&apos;re working on something that someone else will need to sign off on, loop them in before you start. &quot;I&apos;m going to do this thing. Here&apos;s my plan. Any concerns?&quot; Then, at the midpoint, show them the architecture — not the implementation details, just the structure. Names, boundaries, where things live. Get that directional feedback when it&apos;s cheap to act on. Then finish the implementation and let the PR be the final confirmation: here&apos;s what we talked about, here&apos;s what I built, here&apos;s the test coverage that says it works.&lt;/p&gt;
&lt;p&gt;If the PR fails automated checks (coverage below threshold, build broken, linting issues) it doesn&apos;t even need to get to a human. It goes back to the engineer with a clear signal. Fix the basics first. Don&apos;t ask me to spend time on work that doesn&apos;t meet the baseline.&lt;/p&gt;
&lt;p&gt;PRs as the ends of conversations. It&apos;s a small mental shift with a surprisingly large impact on how smoothly a team moves.&lt;/p&gt;
</content:encoded></item><item><title>More Code, More Risk ... More Automation</title><link>https://clintparker.com/blog/more-code-more-risk-more-automation/</link><guid isPermaLink="true">https://clintparker.com/blog/more-code-more-risk-more-automation/</guid><description>AI-assisted development raises output, so validation — fast builds, coverage gates, AI in the testing loop — has to scale with it.</description><pubDate>Sun, 22 Mar 2026 19:11:34 GMT</pubDate><content:encoded>&lt;p&gt;One of the things I keep saying to engineering teams right now is: this tool should improve the process on &lt;em&gt;all&lt;/em&gt; fronts. Not just the building. All of it.&lt;/p&gt;
&lt;p&gt;AI-assisted development is genuinely remarkable. Engineers are shipping features faster, clearing backlogs that felt immovable, and getting real leverage from tooling that would&apos;ve taken months to build before. I&apos;m a fan. I&apos;m actively encouraging it.&lt;/p&gt;
&lt;p&gt;But faster output without better validation is just faster bugs. I&apos;ve seen it — and heard it from peers managing teams at scale. Engineers trust the output, skip the second look, and something breaks in production. The specific failure mode varies. The root cause almost never does: the code moved faster than the confidence in it.&lt;/p&gt;
&lt;p&gt;The issue isn&apos;t that AI makes mistakes. It does, and so do humans. The issue is that when the &lt;em&gt;volume&lt;/em&gt; of work goes up, the &lt;em&gt;surface area for mistakes&lt;/em&gt; goes up with it. If your validation process doesn&apos;t scale at the same rate, you end up cleaning up behind yourself constantly.&lt;/p&gt;
&lt;p&gt;The answer isn&apos;t to slow down. It&apos;s to automate your safety net.&lt;/p&gt;
&lt;p&gt;Fast builds. Fast test runs. Automated coverage thresholds that fail a PR before it ever hits a human reviewer. If the test coverage isn&apos;t there, the PR doesn&apos;t get reviewed. Period. That sounds harsh, but it&apos;s actually generous — it tells the engineer clearly and immediately what they need to do before asking anyone else to spend time on their work.&lt;/p&gt;
&lt;p&gt;I also think there&apos;s something important here about how we use AI in the validation loop, not just the construction loop. Use it to generate test cases. Use it to review test coverage for gaps. Have it challenge your assumptions about how a feature might fail. A lot of what a good QA person does — thinking adversarially about software — can be augmented with the same tools you&apos;re using to write the code.&lt;/p&gt;
&lt;p&gt;The teams that win with AI aren&apos;t the ones moving fastest. They&apos;re the ones who figured out how to move fast &lt;em&gt;and&lt;/em&gt; land cleanly.&lt;/p&gt;
</content:encoded></item><item><title>Testing Is a Team Sport</title><link>https://clintparker.com/blog/testing-is-a-team-sport/</link><guid isPermaLink="true">https://clintparker.com/blog/testing-is-a-team-sport/</guid><description>Separating building from quality creates the distance where bugs hide; the people accountable for delivery must also be accountable for correctness.</description><pubDate>Fri, 20 Mar 2026 17:29:54 GMT</pubDate><content:encoded>&lt;p&gt;I&apos;ve had some version of this conversation with multiple engineering leaders lately, and I keep landing in the same place: &lt;strong&gt;quality is a team sport, and most teams haven&apos;t figured that out yet.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;Here&apos;s the dynamic I see a lot. You&apos;ve got developers who write their code, write their tests (maybe), and hand the whole thing off. Somewhere downstream, a tester or QA person picks it up and starts poking holes. When they find something, it goes back. When they don&apos;t, it ships. Everyone&apos;s doing their job. And yet somehow, bugs still make it to production, morale is still low, and the whole thing feels slower than it should.&lt;/p&gt;
&lt;p&gt;The problem isn&apos;t the people. The problem is the model.&lt;/p&gt;
&lt;p&gt;Separating &quot;building&quot; from &quot;quality&quot; creates distance — between the person who knows the most about how something was constructed and the process of verifying that it works correctly. That distance is where bugs hide.&lt;/p&gt;
&lt;p&gt;The fix isn&apos;t to hire more testers. It&apos;s to collapse that distance. Make the people accountable for delivery also accountable for correctness. Pair them up. If two engineers worked on a feature together, they&apos;re both on the hook if it breaks. Not &quot;well, they were supposed to test it.&quot; No. Both of them own it.&lt;/p&gt;
&lt;p&gt;What I&apos;ve seen work is building that expectation into the culture from the top. Not as a punitive thing, but as a shared ownership thing. &lt;em&gt;&lt;em&gt;We&lt;/em&gt;&lt;/em&gt; built it. &lt;em&gt;&lt;em&gt;We&lt;/em&gt;&lt;/em&gt; stand behind it. If it&apos;s wrong, we fix it, we learn from it, and we make sure it doesn&apos;t happen that way again.&lt;/p&gt;
&lt;p&gt;The corollary to this is that no single person (or department) in an org should carry &quot;quality&quot; as their sole responsibility. One QA lead against a team of fifteen engineers is not a quality strategy. It&apos;s a bottleneck with a job title. The engineerings and their leadership (the people accountable for delivery) also have to be accountable for what they deliver working correctly. That&apos;s not a radical idea. It&apos;s just engineering maturity.&lt;/p&gt;
&lt;p&gt;Quality isn&apos;t a department. It&apos;s an expectation.&lt;/p&gt;
</content:encoded></item><item><title>Feature Flags for Fun &amp; Profit</title><link>https://clintparker.com/blog/feature-flags-for-fun-and-profit/</link><guid isPermaLink="true">https://clintparker.com/blog/feature-flags-for-fun-and-profit/</guid><description>Why we use feature flags as a deployment and risk-mitigation tool by default, with C# implementation patterns and a real-world refactoring example.</description><pubDate>Tue, 11 Mar 2025 18:16:00 GMT</pubDate><content:encoded>&lt;p&gt;&lt;em&gt;By Gerald Singleton &amp;amp; Clint Parker&lt;/em&gt;&lt;/p&gt;
&lt;h2&gt;Intro&lt;/h2&gt;
&lt;p&gt;Continuous integration &amp;amp; deployment. Automated testing. Refactoring &amp;amp; taking control of the monolith. Reducing cycle time. Increased uptime. Optimizing the data layer. Putting stakeholders in control. Making customers happy.&lt;/p&gt;
&lt;p&gt;We’ve done all that. And it was all easier with our aggressive adoption of feature flags. Using flags by default has unblocked all of the more visible initiatives we wanted to achieve. In this document, we’ll showcase what flags are (and aren’t), why we use them to the extent we do, and how you can quickly take advantage of these patterns to improve your codebases.&lt;/p&gt;
&lt;h2&gt;Guidelines&lt;/h2&gt;
&lt;h3&gt;Not (just) for features&lt;/h3&gt;
&lt;p&gt;A common misconception is that feature flags are only for toggling application features. Development teams often tend to think that an application will have a finite number of features at any given time, and those features can be managed via configuration settings. This thinking limits the overall flexibility of your application. In our SDLC, we don&apos;t use feature flags just to toggle application features. In fact, we primarily use feature flags as a deployment tool. They enable us to deploy changes to our product frequently, purposefully, and most importantly…safely.&lt;/p&gt;
&lt;h3&gt;Columns vs. Layers&lt;/h3&gt;
&lt;p&gt;A foundational concept of feature flags is context vs config. Many applications use a variety of pre-production environments with their own config settings and slightly different expectations. An example would be connection strings for the different versions of the same DB. Or maybe a sandbox vs production integration dependencies. These are usually diagramed as vertical, separated by horizontal lines as layers, hence the phrase “lower environments.” Layers are expensive. There are tricks to reduce costs, but nevertheless, they are expensive.&lt;/p&gt;
&lt;p&gt;On the other hand, horizontal segmentations, or columns, tend to be much less expensive and dynamic. These are thought of as usage characteristics, like the number of users, geography, and account settings, which are cheap. For this purpose, we can group all the varieties of horizontal segmentation into the term “context.” Contexts don’t have to have names, nor do they need special dependencies. Contexts are unmanaged. They simply exist within the runtime space of the code.&lt;/p&gt;
&lt;h3&gt;Context&lt;/h3&gt;
&lt;p&gt;Context is not a reserved word in this case; it&apos;s a concept. LaunchDarkly does have a specific meaning, but that’s not what this is. As mentioned, context already exists in your application. You already have settings for the application itself, organizations, users, geographies, and time/date. Contexts can be split or combined. The point is that your application can accommodate this perspective and already does. Accepting this is important because the appropriate usage of feature flags will maximize it.&lt;/p&gt;
&lt;p&gt;Feature flags are just additional context. You should pick one to three known contexts to start with. In a B2B application, the first context you should accommodate in your flagging system is “Company ID.” The new context is flag state plus company.
The myth of tech debt
I’m sure you’re asking yourself, “But aren’t you just increasing the technical debt in the system by adding temporary code?” The short answer is…not really. In our experience, when managed correctly, this is not a problem. When feature flags are implemented in a way in which they can be easily removed (i.e. simple conditional statements), they can (and should) be the shortest-lived code sections in the repo. With this in mind, feature flags can actually be a tool that can be used by the development team to reduce overall technical debt. Armed with this new tool, contributors are empowered to improve the code base aggressively.&lt;/p&gt;
&lt;h3&gt;Risk mitigation&lt;/h3&gt;
&lt;p&gt;Feature flagging’s top value is as a risk mitigator. They can be used for all sorts of other things, but this should be the top priority. If you have such an amazing codebase that you prefer to fly without this safety net, congratulations, you’ve found the perfect engineering shop and should never leave!!! But in most of the engineering teams I’ve been on this hasn’t been the case. Engineers are usually working in code that has been through several development teams with varying levels of skill and backgrounds. This results in a codebase that is often brittle. This is where proper usage of feature flags can shine. Think of how inexpensive it is to add a flag to mitigate any potential risk to your application. Any side effects can be quickly reverted back to the original behavior with the simple flip of a feature flag.&lt;/p&gt;
&lt;p&gt;With the safety net of feature flags in place, you can do the unthinkable…test in production. Feature flags enable you to pick a context (user, company, etc.) and test that hypothesis in a real production environment. The impact would be limited to that context only, and if not, the impact can be quickly disabled without impacting the rest of the team and their deliveries.&lt;/p&gt;
&lt;h3&gt;Refactors unleashed&lt;/h3&gt;
&lt;p&gt;With the safety of feature flags, you can take bigger swings especially as it relates to refactoring code. This proves especially beneficial in older code bases where the system may be more brittle. You can refactor a whole vertical slice, deploying frequently and confirming along the way that the change is working in production with little to no impact to your users. If you find that a piece of refactored code that was deployed causes unintended side effects, you don’t have to go through the process of rushing to make a code fix and redeploying your code to production. It’s as simple as turning off the feature flag. You can take the time to mindfully fix the issue and continue refactoring.&lt;/p&gt;
&lt;h3&gt;Quality mindset / User experience first&lt;/h3&gt;
&lt;p&gt;Since nothing is now stopping the team from improving the system, quality, and intolerance for degraded user experiences can become the norm. Imagine a world where rapid delivery of value to your users is possible without the historical fear of unintended downtime. Part of the SDLC now involves verifying assumptions in production, constantly maintaining the system, and never disappointing your users. Feature flags can literally let you “swap the engine while driving down the freeway.”&lt;/p&gt;
&lt;h3&gt;Aggressive/Liberal usage&lt;/h3&gt;
&lt;p&gt;With all of these benefits, why not aggressively apply feature flags across your code base? The reality is that if the implementation of feature flag usage described here is going to be successful, it must become an enforced standard. It must be a requirement on pull requests that the changes be behind a feature flag. Why does it require that level of enforcement to be successful? Because change is hard, and this change in particular, requires a mindset shift that may not be easy for some engineers. In the same way, writing tests, creating documentation, or following coding standards doesn’t come easy initially.&lt;/p&gt;
&lt;h2&gt;Implementation examples and guidelines&lt;/h2&gt;
&lt;p&gt;Identify your flags not just by the change but also by the team and implementer. Start with a common context for your application. In B2B, the context should be the organization identifier. In B2C, geography is a great place to start.&lt;/p&gt;
&lt;p&gt;To make sure that this point gets across, we want to repeat: “Feature flags are meant to have a short lifespan.” Ensuring that feature flags are temporary is one of the foundations of implementing the strategies outlined here.  Ignoring this foundational topic could lead to situations where you have nested feature flag implementations.  This can severely reduce the maintainability of the code base.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;public class SampleClass : ISampleClass
{
    private readonly IFeatureFlagProvider _featureFlagProvider;

    public SampleClass(IFeatureFlagProvider featureFlagProvider)
    {
        _featureFlagProvider = featureFlagProvider;
    }

    public async Task DoSomething(string inputValue)
    {
        if (_featureFlagProvider.IsEnabled(FeatureFlagEnums.FeatureFlag1)
        {
            if (_featureFlagProvider.IsEnabled(FeatureFlagEnums.FeatureFlag2)
            {
                /// Some Code Here 
            }
            else
            {
                /// Some Code Here 
            }
        }
        else
        {

        }
    }
}
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Figure 1.1 Nested Feature Flags&lt;/p&gt;
&lt;p&gt;One of the issues that we encountered pretty quickly when implementing feature flags was merge conflicts. In our initial implementation the FeatureFlags were defined in a single enum class. To solve this issue, the enum class was split into several partial classes with each developer having their own Enum file (FeatureFlagEnums.Dev1.cs,FeatureFlagEnums.Dev2.cs, etc). Within that class file is a partial declaration to the FeatureFlagEnums class where each developer can list the feature flags that they are working on. This gives the developer compile time notifications of potential conflicts.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;/// Sample service class 
public class CompanyService : ICompanyService
{
    public async Task&amp;lt;string&amp;gt; DoSomethingCool(string inputValue1)
    {
        //Imagine some code here 
    }

    public async Task&amp;lt;string&amp;gt; DoSomethingCooler(string inputValue1)
    {
        //Imagine some cooler code here 
    }
}

/// This is a context object
/// It would be used to pass into the feature flag client
public class FeatureFlagContext : IFeatureFlagContext
{
    public string CompanyId { get; set; }
}

/// This class would represent your feature flag
/// management.  This could be a wrapper around an external feature flag
/// management client such as LaunchDarkly
public class FeatureFlagClient : IFeatureFlagClient
{
    public bool IsFeatureFlagEnabledForContext(string contextId, FeatureFlagEnum featureFlag)
    {
        // Call to your external flag manager here to 
        // retrieve the flag state for the given context
    }
}

/// Pseudo code implementation of the feature flag provider class 
/// This IFeatureFlagContext would contain 
public class FeatureFlagProvider : IFeatureFlagProvider
{
    private readonly IFeatureFlagContext _featureFlagContext;
    private readonly IFeatureFlagClient _featureFlagClient;

    public FeatureFlagProvider(IFeatureFlagContext featureFlagContext, IFeatureFlagClient featureFlagClient)
    {
        //The context could be the HttpContext of the session 
        // (HttpContext.Current) or some other context object.
        // The provider will need to account for if your  
        // context object is null and return the appropriate value
        // from the IsEnabled property
        _featureFlagContext = featureFlagContext;
        _featureFlagClient = featureFlagClient;
    }

    public bool IsEnabled(FeatureFlagEnum featureFlag)
    {
        /// Use the context to determine whether the feature is turned on for the specified context       
        if (_featureFlagContext.CompanyId == null)
        {
            return false;
        }
        else
        {
            return _featureFlagClient.IsFeatureFlagEnabledForContext(_featureFlagContext.CompanyId, featureFlag);
        }
    }
}

//Once compiled both of these feature flags will be part of the FeatureFlagsEnum object

/// &amp;lt;summary&amp;gt;
/// This specific file belongs to: FeatureFlagsEnum.Dev1.cs
/// &amp;lt;/summary&amp;gt;
public static partial class FeatureFlagEnums
{
    public const string InternalBugFixIssueFlag = &quot;internal-31119-sample-feature-flag&quot;;
}

/// &amp;lt;summary&amp;gt;
/// This specific file belongs to: FeatureFlagEnums.Dev2.cs
/// &amp;lt;/summary&amp;gt;
public static partial class FeatureFlagEnums
{
    public const string Company321BugFixIssue = &quot;internal-40101-sample-feature-flag&quot;;
}

//Use of the feature flag in code would look like this
public class FooService : IFooService
{
    private readonly IFeatureFlagProvider _featureFlagProvider;
    private readonly ICompanyService _companyService;


    public FooService(IFeatureFlagProvider featureFlagProvider, ICompanyService companyService)
    {
        _featureFlagProvider = featureFlagProvider;
        _companyService = companyService;
    }


    public async Task&amp;lt;string&amp;gt; DoFoo(string inputValue)
    {
        if (_featureFlagProvider.IsEnabled(FeatureFlagEnums.InternalBugFixIssueFlag))
        {
            return await _companyService.DoSomethingCool(inputValue);
        }
        else
        {
            return await _companyService.DoSomethingCooler(inputValue);
        }
    }
}
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Figure 1.2 Initial Introduction of the feature flag into code.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;//Foo service once the feature flag has been removed
public class FooService : IFooService
{
    private readonly IFeatureFlagProvider _featureFlagProvider;
    private readonly ICompanyService _companyService;

    public Foo(IFeatureFlagProvider featureFlagProvider, ICompanyService companyService)
    {
        _featureFlagProvider = featureFlagProvider;
        _companyService = companyService;
    }

    public async Task&amp;lt;string&amp;gt; DoFoo(string inputValue)
    {
        return await _companyService.DoSomethingCool(inputValue);
    }
}


/// Unused code removed from the company service
public class CompanyService : ICompanyService
{
    public async Task&amp;lt;string&amp;gt; DoSomethingCooler(string inputValue1)
    {
        //Imagine some cooler code here 
    }


    public async Task&amp;lt;string&amp;gt; DoTheCoolestThing(string inputValue1)
    {
        //Imagine some of the coolest code here 
    }
}
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Figure 1.3 Same code block from Figure 1.1 after the feature flag has been removed.
Real World Scenarios&lt;/p&gt;
&lt;h3&gt;The Big Refactor&lt;/h3&gt;
&lt;p&gt;If your development team is not made up of AI agents yet, then you’ve heard the phrase “I want to rewrite that whole feature from scratch”.  In one of the development teams I worked with, we wanted to remove the use of stored procedures and replace it with an object relational mapping tool (ORM).  On the surface it doesn’t sound crazy, until you factor in that the application had over 900+ stored procedures.   Where in most development organizations this would be a non-starter, we were able to start work on this immediately.  How? Look at Figure 2.1 to see where we started.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;/// Parameter Values Class 
public class ParameterValue
{
    public string ParameterName { get; set; }
    public object ParamterValue { get; set; }

    public ParameterValue(string parameterName, object parameterValue)
    {
        ParameterName = parameterName;
        ParameterValue = parameterValue;
    }
}

/// Generic Data Service
public class DataService : IDataService
{
    public async Task GenericQuery1(string inputValue)
    {
        ExecuteStoredProcedure(&quot;GenericStoredProcedure&quot;,
            new List&amp;lt;ParameterValue&amp;gt;(){
                new ParameterValue(&quot;Value1&quot;, inputValue)
            });
    }

    public async Task GenericQuery2(string inputValue)
    {
        ExecuteStoredProcedure(&quot;AnotherStoredProcedure&quot;,
            new List&amp;lt;ParameterValue&amp;gt;(){
                new ParameterValue(&quot;Value1&quot;, inputValue)
            });
    }

    private async Task ExecuteStoredProcedure(string procedureName, List&amp;lt;ParameterValue&amp;gt; paramters)
    {
        /// Code to Execute Stored Procedure against a data store here 
    }
}
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Figure 2.1 - Before any Changes&lt;/p&gt;
&lt;p&gt;Nothing super interesting in that code snippet.  Your normal boiler plate stored procedure execution.  But look at how easy it was for us to start implementing changes to how we are accessing our data with a few lines of code. Look at figure 2.2&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;/// Generic Data Service
public class DataService : IDataService
{
    private readonly IFeatureFlagProvider _featureFlagProvider;

    public DataService(IFeatureFlagProvider featureFlagProvider)
    {
        _featureFlagProvider = featureFlagProvider;
    }

    public async Task GenericQuery1(string inputValue)
    {
        // Add a feature flag if statement here
        if (_featureFlagProvider.IsEnabled(FeatureFlagsEnum.UseNewQuery1))
        {
            await GetData(
                new List&amp;lt;ParameterValue&amp;gt;(){
                    new ParameterValue(&quot;Value1&quot;, inputValue)
                });
        }
        else
        {
            await ExecuteStoredProcedure(&quot;GenericStoredProcedure&quot;,
             new List&amp;lt;ParameterValue&amp;gt;(){
                new ParameterValue(&quot;Value1&quot;, inputValue)
            });
        }
    }

    public async Task GenericQuery2(string inputValue)
    {
        ExecuteStoredProcedure(&quot;AnotherStoredProcedure&quot;,
            new List&amp;lt;ParameterValue&amp;gt;(){
                new ParameterValue(&quot;Value1&quot;, inputValue)
            });
    }


    /// New method that doesn&apos;t use stored procedures 
    private async Task GetData(List&amp;lt;ParameterValue&amp;gt; parameters)
    {

    }

    private async Task ExecuteStoredProcedure(string procedureName, List&amp;lt;ParameterValue&amp;gt; paramters)
    {
        /// Code to Execute Stored Procedure against a data store here 
    }
}
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Figure 2.2&lt;/p&gt;
&lt;p&gt;Using feature flags we were able to start chipping away at a major refactor while still maintaining the legacy code.  By using a context, we could control how many users were executing our new source code.  We could deploy several changes with little to no user impact.&lt;/p&gt;
</content:encoded></item><item><title>2023 Year in Review</title><link>https://clintparker.com/blog/2023-year-in-review/</link><guid isPermaLink="true">https://clintparker.com/blog/2023-year-in-review/</guid><description>A rundown of the changes my team adopted in 2023: feature flags, hourly deploys, Cloudflare, Blazor WASM, Linux in production, and more.</description><pubDate>Mon, 08 Jan 2024 18:44:36 GMT</pubDate><content:encoded>&lt;p&gt;My team has seen a lot of changes in the last year. These are things that we didn&apos;t really have in 2022 but are became a part of our day-to-day in 2023.&lt;/p&gt;
&lt;h3&gt;Feature flags&lt;/h3&gt;
&lt;p&gt;We started to introduce the concept of flags in late 2022 but didn&apos;t adopt them until 2023. We&apos;ve rewritten the framework a few times. The team has created guidelines for flag creation, management, and removal. We&apos;ve introduced over 200 flags in 2023. The adoption of our feature flag process has led to...&lt;/p&gt;
&lt;h3&gt;Deploying multiple times per day&lt;/h3&gt;
&lt;p&gt;In May of 2023, we moved to hourly deploys. We had previously been on a structured 2-week deployment cadence. There are some specific challenges with a 2-week cadence: maintaining the &quot;release branch,&quot; being beholden to the release schedule and work done or not done in time, the fact that we were deploying a bundle of 2 weeks of work, and hotfixes bypassing all the process. We&apos;ve since moved to hourly deploys. We currently deploy on the hour and will be moving to full continuous deployment in January. Production incident remediation times are now tracked in minutes and not hours.&lt;/p&gt;
&lt;h3&gt;DDOS protection&lt;/h3&gt;
&lt;p&gt;In 2023, we moved our WAF to Cloudflare. This has given us DDOS protection and a CDN. The DDOS mitigation has proved extremely valuable, as our system has been able to withstand attacks over 10M requests per minute.&lt;/p&gt;
&lt;h3&gt;WASM&lt;/h3&gt;
&lt;p&gt;We&apos;ve introduced Blazor to our stack to add frontend code quickly and reliably. We&apos;re using Blazor WASM, which is C# and HTML compiled to WebAssemly. This allows us to use our C# knowledge and best practices (including automated testing) for browser code.&lt;/p&gt;
&lt;h3&gt;Running on Linux in prod&lt;/h3&gt;
&lt;p&gt;In the first half of 2023, we migrated our production servers to Linux. In the second half of the year, we migrated our remaining dev and staging servers to Linux. We&apos;ve also migrated our build servers to Linux. These migrations saved costs on the computing side, allowing us to scale up our data side without any overall cost increase.&lt;/p&gt;
&lt;h3&gt;Latest .NET&lt;/h3&gt;
&lt;p&gt;Staying on the latest version of the framework is uncommon in most .NET shops. In 2022, we migrated to dotnet 6. In 2023, we&apos;ve done it again and migrated to dotnet 7. In early 2024, we&apos;ll move to the newly released dotnet 8.&lt;/p&gt;
&lt;h3&gt;Increased automated testing&lt;/h3&gt;
&lt;p&gt;In August, we increased our expectations around automated testing. We&apos;re now near 40% for total line coverage for all codebases. We&apos;ve adopted behavioral testing across all of the backend code. We&apos;ve introduced Playwright, which allows us to test our frontend code in a more automated fashion.&lt;/p&gt;
&lt;h3&gt;Codified SDLC&lt;/h3&gt;
&lt;p&gt;In 2022, our SDLC was very loose and ad-hoc. In 2023, we&apos;ve codified our SDLC. Our SDLC is meant to be flexible while maintaining consistency across the department. Our SDLC guidelines represent sensible defaults, and we hope they will continue to evolve to best serve the teams leveraging them.&lt;/p&gt;
&lt;h3&gt;Structured teams&lt;/h3&gt;
&lt;p&gt;At the end of 2023, we had one team of 12, one team of 5, and one team of 2 with QA floating across teams. We&apos;ve since restructured into 3 teams of even size and even staffing.&lt;/p&gt;
&lt;h3&gt;Job descriptions&lt;/h3&gt;
&lt;p&gt;I know the engineering team had been working on some job descriptions/matrices, but they never quite made it to fruition. This year, Engineering leadership created measurable job expectations for software engineering levels 1-4. We&apos;ve published these to our team and are using them in our 1:1s and reviews. This gives clarity to both our team members and managers. We&apos;ll be creating similar documents for our managers and QA and DevOps teams in 2024.&lt;/p&gt;
&lt;h3&gt;Consistent meeting schedule&lt;/h3&gt;
&lt;p&gt;In addition to the meeting guidelines of our SDLC, we&apos;ve also established a monthly department-wide meeting. This meeting is an opportunity to showcase the great work done each month, share department-level information, and keep each other accountable for our organizational goals.&lt;/p&gt;
&lt;h3&gt;Company-wide bug reporting&lt;/h3&gt;
&lt;p&gt;Open bug reporting is a sign of engineering team maturity, and in May of 2023, we opened up our bug reporting process to the whole company. We previously had two competing processes. Not only did this reduce transparency and create confusion, but issues reported in the support team&apos;s system had to be verified and triaged before being added to the engineering backlog. This dual process limited visibility into the bug backlog and also skewed reporting.&lt;/p&gt;
&lt;p&gt;This has been one of the most remarkable years of my career. Teams rarely see this much evolution in such a short time. I can&apos;t wait to see what interesting enhancements 2024 delivers.&lt;/p&gt;
</content:encoded></item><item><title>Improving Software Team Metrics</title><link>https://clintparker.com/blog/improving-software-team-metrics/</link><guid isPermaLink="true">https://clintparker.com/blog/improving-software-team-metrics/</guid><description>Rolling sprint completion percentages up to monthly values, and four ways to aggregate them across teams — including an inverted calculation that surfaces missed commitments.</description><pubDate>Mon, 27 Nov 2023 19:30:46 GMT</pubDate><content:encoded>&lt;p&gt;A healthy engineering organization (or any healthy team, for that matter) should be tracking itself across a variety of metrics. This is not covered by the standard CS curriculum but is readily encountered in the real world. Once someone is paying for software, there will invariably be questions about how that money is being spent. The most common metrics are bug count and velocity. Followed by automated code coverage. These are common because they&apos;re the cheapest to produce. Bugs are, unfortunately, the most visible part of engineering output. Counting them is the start of reducing them. Code coverage is freely available in every modern build pipeline, although not always enabled. And velocity is the treasured metric of any young engineering leader, the end-all answer to the question “How much work are we getting done!?&quot;&lt;/p&gt;
&lt;p&gt;However, once you start looking, there is so much more insight you can gain and so many more things to track and compare. And, eventually, when you&apos;re answering to very clever investors, you&apos;ll need to provide the metrics that they care about. One of those, which I have come to appreciate, is the sprint completion percentage. This expounds on velocity and compares that &lt;em&gt;actual&lt;/em&gt; value to the estimated or planned value. A high velocity is excellent, but accurate forecasting is even better for the overall business. This metric is easy enough to retrieve. Azure DevOps (ADO) has this baked into its velocity dashboards. The granularity is obviously at the sprint level.&lt;/p&gt;
&lt;p&gt;With a little API magic, we can easily get:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Team&lt;/th&gt;
&lt;th&gt;Iteration Path&lt;/th&gt;
&lt;th&gt;StartDate&lt;/th&gt;
&lt;th&gt;EndDate&lt;/th&gt;
&lt;th&gt;Planned&lt;/th&gt;
&lt;th&gt;Completed&lt;/th&gt;
&lt;th&gt;Completed Late&lt;/th&gt;
&lt;th&gt;Incomplete&lt;/th&gt;
&lt;th&gt;Total&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Avengers&lt;/td&gt;
&lt;td&gt;21&lt;/td&gt;
&lt;td&gt;2023-10-10&lt;/td&gt;
&lt;td&gt;2023-10-23&lt;/td&gt;
&lt;td&gt;87&lt;/td&gt;
&lt;td&gt;58&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;58&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Avengers&lt;/td&gt;
&lt;td&gt;20&lt;/td&gt;
&lt;td&gt;2023-09-26&lt;/td&gt;
&lt;td&gt;2023-10-09&lt;/td&gt;
&lt;td&gt;46&lt;/td&gt;
&lt;td&gt;38&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;38&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Avengers&lt;/td&gt;
&lt;td&gt;19&lt;/td&gt;
&lt;td&gt;2023-09-12&lt;/td&gt;
&lt;td&gt;2023-09-25&lt;/td&gt;
&lt;td&gt;51&lt;/td&gt;
&lt;td&gt;50&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;50&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;X-Men&lt;/td&gt;
&lt;td&gt;21&lt;/td&gt;
&lt;td&gt;2023-10-10&lt;/td&gt;
&lt;td&gt;2023-10-23&lt;/td&gt;
&lt;td&gt;51&lt;/td&gt;
&lt;td&gt;41&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;41&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;X-Men&lt;/td&gt;
&lt;td&gt;20&lt;/td&gt;
&lt;td&gt;2023-09-26&lt;/td&gt;
&lt;td&gt;2023-10-09&lt;/td&gt;
&lt;td&gt;66&lt;/td&gt;
&lt;td&gt;79&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;79&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;X-Men&lt;/td&gt;
&lt;td&gt;19&lt;/td&gt;
&lt;td&gt;2023-09-12&lt;/td&gt;
&lt;td&gt;2023-09-25&lt;/td&gt;
&lt;td&gt;18&lt;/td&gt;
&lt;td&gt;30&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;30&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Justice League&lt;/td&gt;
&lt;td&gt;21&lt;/td&gt;
&lt;td&gt;2023-10-10&lt;/td&gt;
&lt;td&gt;2023-10-23&lt;/td&gt;
&lt;td&gt;90&lt;/td&gt;
&lt;td&gt;75&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;75&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Justice League&lt;/td&gt;
&lt;td&gt;20&lt;/td&gt;
&lt;td&gt;2023-09-26&lt;/td&gt;
&lt;td&gt;2023-10-09&lt;/td&gt;
&lt;td&gt;120&lt;/td&gt;
&lt;td&gt;121&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;129&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Justice League&lt;/td&gt;
&lt;td&gt;19&lt;/td&gt;
&lt;td&gt;2023-09-12&lt;/td&gt;
&lt;td&gt;2023-09-25&lt;/td&gt;
&lt;td&gt;108&lt;/td&gt;
&lt;td&gt;77&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;77&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;The definitions for these states can be found &lt;a href=&quot;https://learn.microsoft.com/en-us/azure/devops/report/dashboards/team-velocity?view=azure-devops&amp;amp;tabs=in-context#state-descriptions-table&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;We need to do a little more math, though, for this to become a valuable reporting metric. Unfortunately, the rest of the business and the investors don&apos;t care about your sprints; they care about monthly and quarterly aggregates.&lt;/p&gt;
&lt;p&gt;So, let&apos;s start there with the math that rolls up sprints to a monthly value. It&apos;s pretty fun. We need to determine what month a sprint falls into. My calculation chooses the month that contains more days of the sprint, and if it is equal, the sprint starts.&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Team&lt;/th&gt;
&lt;th&gt;Iteration Path&lt;/th&gt;
&lt;th&gt;StartDate&lt;/th&gt;
&lt;th&gt;EndDate&lt;/th&gt;
&lt;th&gt;Planned&lt;/th&gt;
&lt;th&gt;Completed&lt;/th&gt;
&lt;th&gt;Completed Late&lt;/th&gt;
&lt;th&gt;Incomplete&lt;/th&gt;
&lt;th&gt;Total&lt;/th&gt;
&lt;th&gt;Completion %&lt;/th&gt;
&lt;th&gt;Month&lt;/th&gt;
&lt;th&gt;Year&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Avengers&lt;/td&gt;
&lt;td&gt;21&lt;/td&gt;
&lt;td&gt;2023-10-10&lt;/td&gt;
&lt;td&gt;2023-10-23&lt;/td&gt;
&lt;td&gt;87&lt;/td&gt;
&lt;td&gt;58&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;58&lt;/td&gt;
&lt;td&gt;67%&lt;/td&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;2023&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Avengers&lt;/td&gt;
&lt;td&gt;20&lt;/td&gt;
&lt;td&gt;2023-09-26&lt;/td&gt;
&lt;td&gt;2023-10-09&lt;/td&gt;
&lt;td&gt;46&lt;/td&gt;
&lt;td&gt;38&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;38&lt;/td&gt;
&lt;td&gt;83%&lt;/td&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;2023&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Avengers&lt;/td&gt;
&lt;td&gt;19&lt;/td&gt;
&lt;td&gt;2023-09-12&lt;/td&gt;
&lt;td&gt;2023-09-25&lt;/td&gt;
&lt;td&gt;51&lt;/td&gt;
&lt;td&gt;50&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;50&lt;/td&gt;
&lt;td&gt;98%&lt;/td&gt;
&lt;td&gt;9&lt;/td&gt;
&lt;td&gt;2023&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;X-Men&lt;/td&gt;
&lt;td&gt;21&lt;/td&gt;
&lt;td&gt;2023-10-10&lt;/td&gt;
&lt;td&gt;2023-10-23&lt;/td&gt;
&lt;td&gt;51&lt;/td&gt;
&lt;td&gt;41&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;41&lt;/td&gt;
&lt;td&gt;80%&lt;/td&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;2023&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;X-Men&lt;/td&gt;
&lt;td&gt;20&lt;/td&gt;
&lt;td&gt;2023-09-26&lt;/td&gt;
&lt;td&gt;2023-10-09&lt;/td&gt;
&lt;td&gt;66&lt;/td&gt;
&lt;td&gt;79&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;79&lt;/td&gt;
&lt;td&gt;120%&lt;/td&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;2023&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;X-Men&lt;/td&gt;
&lt;td&gt;19&lt;/td&gt;
&lt;td&gt;2023-09-12&lt;/td&gt;
&lt;td&gt;2023-09-25&lt;/td&gt;
&lt;td&gt;18&lt;/td&gt;
&lt;td&gt;30&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;30&lt;/td&gt;
&lt;td&gt;167%&lt;/td&gt;
&lt;td&gt;9&lt;/td&gt;
&lt;td&gt;2023&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Justice League&lt;/td&gt;
&lt;td&gt;21&lt;/td&gt;
&lt;td&gt;2023-10-10&lt;/td&gt;
&lt;td&gt;2023-10-23&lt;/td&gt;
&lt;td&gt;90&lt;/td&gt;
&lt;td&gt;75&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;75&lt;/td&gt;
&lt;td&gt;83%&lt;/td&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;2023&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Justice League&lt;/td&gt;
&lt;td&gt;20&lt;/td&gt;
&lt;td&gt;2023-09-26&lt;/td&gt;
&lt;td&gt;2023-10-09&lt;/td&gt;
&lt;td&gt;120&lt;/td&gt;
&lt;td&gt;121&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;129&lt;/td&gt;
&lt;td&gt;108%&lt;/td&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;2023&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Justice League&lt;/td&gt;
&lt;td&gt;19&lt;/td&gt;
&lt;td&gt;2023-09-12&lt;/td&gt;
&lt;td&gt;2023-09-25&lt;/td&gt;
&lt;td&gt;108&lt;/td&gt;
&lt;td&gt;77&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;77&lt;/td&gt;
&lt;td&gt;71%&lt;/td&gt;
&lt;td&gt;9&lt;/td&gt;
&lt;td&gt;2023&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Aggregating these values can be done in a few different ways. We&apos;re combining teams and sprints to get a monthly representation for the group as a whole. I&apos;ve found four reasonable ways to calculate this value across teams and sprints:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Basic Average&lt;/li&gt;
&lt;li&gt;Unweighted Average&lt;/li&gt;
&lt;li&gt;Weighted Average&lt;/li&gt;
&lt;li&gt;&quot;Inverted&quot;&lt;/li&gt;
&lt;/ul&gt;
&lt;h4&gt;Basic Average&lt;/h4&gt;
&lt;p&gt;The most basic average. This would be the average of all the values for the &lt;code&gt;Completion %&lt;/code&gt; column for a given month and year. While this is a straightforward value to calculate, I&apos;ve found it gives too much weight to the individual sprints. For example, one lousy sprint, even with a minimal planned value, can drastically change this calculation.&lt;/p&gt;
&lt;h4&gt;Unweighted&lt;/h4&gt;
&lt;p&gt;This is the sum of the &lt;code&gt;Total&lt;/code&gt; column divided by the sum of the &lt;code&gt;Planned&lt;/code&gt; column for a given month and year. This assigns too little weight to individual sprints and doesn&apos;t address the discrepancies in point values across teams.&lt;/p&gt;
&lt;h4&gt;Weighted&lt;/h4&gt;
&lt;p&gt;This has been my go-to calculation for years. This is a two-phased calculation. First, we roll up the value for the individual teams. We do this with the unweighted model but filter by &lt;code&gt;Team&lt;/code&gt; in addition to month and year. Then, we average those values. This handles a team having a lousy sprint but recovering in the next, as well as the differences in point values.&lt;/p&gt;
&lt;p&gt;But what about team B? They didn&apos;t get all that work done. It doesn&apos;t feel like the numbers represent the reality if the work not getting done was high value / high vis. The 1st phase of the weighted model allows for a disappointing sprint. And if the team is working ahead or catching up, we&apos;re sweeping that bad sprint under the rug. While this hadn&apos;t always directly worried me, my colleagues who had been expecting certain things and not seeing them delivered despite the 100%+ completion rates were getting a little frustrated.&lt;/p&gt;
&lt;p&gt;So I&apos;ve come up with a new number to properly represent just that: how much work &lt;em&gt;we aren&apos;t&lt;/em&gt; getting done every month.&lt;/p&gt;
&lt;h4&gt;&quot;Inverted&quot;&lt;/h4&gt;
&lt;p&gt;&quot;Inverted&quot; may be more representative of the commitment to the business. It shows if we did what we committed to but discounts the value of above and beyond work. This calculation has a maximum of 100%. The calculation is multi-phased. The first phase is the same as weighted. Then, we “invert&quot; the monthly team values. If the number is less than 100%, we report the difference; otherwise, we report 0. Next, we average those shortfall percentages. And finally, we subtract that value from 100%.&lt;/p&gt;
&lt;p&gt;The inverted value is more representative of our accountability to the business. It should be noted that this value doesn&apos;t entirely neglect above and beyond work but severely discounts it. Namely, when the X-Men go above and beyond, it won&apos;t outweigh the shortcomings of the Avengers that month.&lt;/p&gt;
&lt;h4&gt;Conclusion&lt;/h4&gt;
&lt;p&gt;Tracking software team metrics is an essential aspect of maintaining a healthy engineering organization. While common metrics such as bug count and velocity provide a basic understanding of team performance, they often fall short in providing a comprehensive view of the team&apos;s efficiency and productivity. This article has explored the concept of sprint completion percentage as a more insightful metric, offering a comparison of actual work done against planned work.&lt;/p&gt;
&lt;p&gt;In essence, the choice of metric and calculation method should align with the team&apos;s objectives and the expectations of stakeholders. By adopting a more nuanced approach to tracking software team metrics, organizations can gain deeper insights into team performance, improve forecasting accuracy, and ultimately drive better business outcomes.&lt;/p&gt;
</content:encoded></item><item><title>What Even Is Innovation?</title><link>https://clintparker.com/blog/what-even-is-innovation/</link><guid isPermaLink="true">https://clintparker.com/blog/what-even-is-innovation/</guid><description>The innovation I&apos;m proudest of: diagnosing a Classic ASP CPU regression and building static analysis of interpreted line counts into the build process.</description><pubDate>Wed, 08 Nov 2023 18:05:08 GMT</pubDate><content:encoded>&lt;p&gt;I was once asked about the most inventive or innovative thing I’d done. Where to start? I’m a middling engineer at best. I fully subscribe to my own pitch as a leader that engineers should prioritize simplicity and obviousness over performance and cleverness.&lt;/p&gt;
&lt;p&gt;That said, I have an obvious answer to &quot;What is the most interesting problem you ever solved?&quot; And just to be transparent and fair, I didn&apos;t solve this in a vacuum. I worked with a great team and would not have succeeded without their help.&lt;/p&gt;
&lt;p&gt;The innovation I&apos;m proud of is a little embarrassing due to the underlying technology. While I was at Mindbody, we uncovered an impactful limitation of scaling Classic ASP web applications. That&apos;s right, Mindbody was still very much reliant on Classic ASP, which had been deprecated with the arrival of &lt;code&gt;.NET&lt;/code&gt;. The solution to this scaling problem wasn&apos;t particularly complex, but the novelty and impact qualify as innovative. In the end, we were able to proactively identify, remediate, and prevent future consequences of the limitation.&lt;/p&gt;
&lt;p&gt;In late 2017, our VP of Engineering asked me to investigate an issue plaguing another team in his org. I was a Senior Manager overseeing other teams in technically a different department, but I and some of my group had historical experience in the code in question. The nominal problem: a deployed bundle of changes resulted in a 10% increase in CPU usage in production. Rolling the deployment back brought the usage back down, and vice versa. Additionally, the CPU increase was not detectable outside of the production environment. ☹️&lt;/p&gt;
&lt;p&gt;I started by enlisting one of the senior engineers on my team, and we began reviewing the changes in the associated deployment. Nothing initially jumped out at us, but on the 3rd pass, I began to suspect that the problem could be related to a change of an &lt;code&gt;#include&lt;/code&gt; reference file. Please see my early post about conditional &lt;code&gt;include&lt;/code&gt; references to understand why this is already a potential issue. (And begin to understand my absolute hat of the continued use of VBScript). -- Side note: VBScript is awesome circa 1997. But, like everything else in the universe, we evolved, and the evolution of VBScript on the server was &lt;code&gt;.NET&lt;/code&gt;. Now, if you want to complain about people &lt;em&gt;choosing&lt;/em&gt; to use VBScript after 2001, I&apos;d be happy to drink my sorrows away beside you. Rant over, for now-- This reference file had &lt;em&gt;itself&lt;/em&gt; added another reference, which is typical. But in this case, the outer file was almost ubiquitously referenced in every top-level file. Specifically, the heavy usage of the modified file meant that this small change was probably causing a wider-than-obvious impact.&lt;/p&gt;
&lt;p&gt;To test the hypothesis that this one-line change was the culprit, we removed that commit from the bundle and redeployed it without issue. The CPU usage increase disappeared! While the immediate problem was solved, I still wanted to know the root cause and prevention methods.&lt;/p&gt;
&lt;p&gt;I then endeavored to prove this issue was detectable via static code analysis. My second hypothesis was that this was related to the server doing more work interpreting more lines per request. The structure of Classic ASP requires that every single line be interpreted when served. Therefore, I suspected that more lines interpreted meant more work being done per request and, in turn, higher CPU usage.&lt;/p&gt;
&lt;p&gt;We created a NodeJS command line tool to analyze the codebase to represent this. We used NodeJS because it truly is the best way to share multi-platform CLIs. And thank you, &lt;a href=&quot;https://github.com/tj&quot;&gt;TJ&lt;/a&gt;, for &lt;code&gt;commander.js&lt;/code&gt;! The references in the &lt;code&gt;include&lt;/code&gt; files created an easily traversed tree. The tree was then flattened and converted to a total number of interpreted lines for any given top-level file.&lt;/p&gt;
&lt;p&gt;We enhanced the tool to provide additional insights, such as the theoretical minimum total lines (fully optimized but impractical to maintain) and the specific references to any included file, as well as a bloat factor, which represented how far the structure of a file was from the optimal. The results were output as one CSV file and a collection of JSON files.&lt;/p&gt;
&lt;p&gt;The results were astounding! The original (problematic) one-line change increased the total number of interpreted lines from 26 million to 52 million. On the other side of the spectrum, the theoretical optimal number of lines was just over 12 million.&lt;/p&gt;
&lt;p&gt;From the insights gleaned from the analysis, we could then restructure the file references to a more optimal state. Finally, we submitted pull requests to the owning team and reduced the total interpreted lines to 19 million.&lt;/p&gt;
&lt;p&gt;Lastly, I saw that this specific issue could be prevented with these new insights. So, we created a step in the build process to run the analysis and limit the total interpreted lines not to exceed a variable maximum value.&lt;/p&gt;
&lt;p&gt;Over the years, other engineers extended the tool to support visualizations of the reference tree and various library upgrades and bug fixes. It was still a critical build step at the time of my departure.&lt;/p&gt;
&lt;p&gt;While none of the technology is particularly glamorous, I am proud of this innovation. Over a few weeks, existing concepts and platforms were reorganized to create something novel and beneficial. We didn&apos;t patent anything. We didn&apos;t write a new language. Heck, we couldn&apos;t even really talk about it for two main reasons: 1. The org didn&apos;t want to admit to using outdated technology, 2. Who else was using that tech and would be interested in listening?&lt;/p&gt;
&lt;p&gt;So, as I said at the beginning, I subscribe to my own pitch of simplicity. We used basic tools and concepts and put them together in a new way.&lt;/p&gt;
&lt;p&gt;P.S. I&apos;m not sure how much we saved the company, but it has to be substantial. At least 10 teams were blocked for 3 weeks from deploying to production. I think they would&apos;ve continued to run into this issue, even if they found it in this instance, and probably would&apos;ve resorted to massively overscaling production infrastructure. Yikes!&lt;/p&gt;
&lt;p&gt;P.P.S. Let&apos;s take a minute to discuss what was probably happening here. I say probably because I don&apos;t know for sure the absolute underlying issue, and even if I did, there really isn&apos;t any fixing it for this ecosystem.&lt;/p&gt;
&lt;p&gt;VBScript works by retrieving the requested page/resource (&lt;code&gt;something.asp&lt;/code&gt;) and then processing the contents based on the context/request and rendering the output. Again, top-notch for 1997.&lt;/p&gt;
&lt;p&gt;VBScript is a v1 product. It isn&apos;t optimized beyond what the engineers fathomed at the time of writing. So, VBScript pulls the initial ASP file from disk and processes it line-by-line. If there is an &lt;code&gt;#include&lt;/code&gt;, it retrieves that and also processes it line-by-line. Why does it process every line? Because it&apos;s a scripting language at heart, and those lines can modify global state outside of a method body( again, see my post on VBScript conditional includes). So, it is doing a lot of work for each page request. The engineers knew about this, so they created a cache of page contents to not have to go to disk every time.&lt;/p&gt;
&lt;p&gt;In our case, though, these two concept collide and clobber each other. The need to process each request creates a ton of work, and the page sizes themselves become massive due to the (substantial but not infinite) recursive nature of the pages. Doing more work, and the cache can&apos;t keep up, so it&apos;s doing more work in vain. Brutal.&lt;/p&gt;
&lt;p&gt;In the end, they did improve Classic ASP/VBScript ... they created &lt;code&gt;.NET&lt;/code&gt;.&lt;/p&gt;
</content:encoded></item></channel></rss>