**A note before you read: I’m a recruiter, not an engineer. This article was inspired by conversations I’ve had with engineering leaders and discussions with my technical leadership team to better understand how different companies are measuring AI activity, evaluating its impact, and thinking about their overall AI readiness. I’m sharing what I’m learning along the way to help other recruiters and talent professionals better understand where organizations are in their AI journey.**
One of the questions I hear as companies adopt AI across software engineering is: How do we know if it’s actually making us more productive?
It’s not exactly an easy question.
I’ve been having conversations with engineering leaders about how they’re measuring AI adoption, and I’m seeing an interesting evolution. Some companies are still measuring AI productivity through the activity happening inside the development process. Others are starting to move the measurement further downstream. That distinction matters because there’s a big difference between producing more software and delivering more value.
The “old way” of measuring AI productivity
When organizations first started adopting AI coding tools, it made sense to measure what was immediately visible. How many pull requests are engineers creating? How frequently are they committing code? How quickly are they completing stories? Did sprint velocity increase? How much code are they producing? How many tokens are they consuming?
The thinking is straightforward. If an engineer used to estimate five hours to complete a story and can now do it in two, AI must be making them more productive. If a team was creating 20 pull requests a month and is now creating 40, output must have increased.
But there’s a problem: a pull request isn’t a product.
More code doesn’t necessarily mean more value. AI makes it easier to generate code, which means metrics based heavily on code volume or development activity can become less meaningful. A team can produce more PRs while also producing more bugs, more rework, or more code that ultimately doesn’t make it into a usable product.
The metric is measuring something real. It just isn’t necessarily measuring what leadership ultimately cares about.
The measurement needs to move
I think the conversation needs to move further in the delivery chain.
Instead of asking, “How much did our engineers produce?” we should be asking, “How much did we actually deliver?”
That means moving from input and activity metrics toward output metrics. Rather than measuring the number of PRs, look at the delivery of larger pieces of functionality. A collection of user stories might make up an epic, and that epic ultimately represents something meaningful the business needs delivered.
Now the measurement becomes: How many features or epics were we delivering before AI? How many are we delivering now? Are those features actually working? Have they been validated? Can the business or customer use them?
That’s a much more meaningful conversation than simply counting the amount of code generated along the way.
One approach our client takes
One firm we support is a good example of where things are headed.
Their approach isn’t simply, “We’re using AI, so our engineers should be faster.” They’re looking at the relationship between AI investment and actual software delivery. That means considering AI tooling costs and token usage alongside the amount of meaningful functionality being delivered.
And that creates a much more interesting measurement.
Imagine an engineering organization previously delivered two meaningful features per month with two engineers. Now those same engineers are using AI. The company is paying for AI tools, consuming tokens, and taking on an additional technology cost.
But suppose the organization is now delivering four meaningful features per month.
Now leadership can ask a question that a PR-counting metric can’t answer: Did the additional investment actually produce additional value?
AI shouldn’t get credit simply because engineers are using it. It should have to demonstrate that the organization is getting something meaningful in return.
The cost comparison matters, too
There’s another question companies should be asking: Would we have gotten more value by investing somewhere else?
If AI adds additional tooling and usage costs, but the organization isn’t delivering materially more software, what exactly did that investment accomplish? Would another engineer have produced more value? Would better development infrastructure have produced more value? Would investing in the engineering process have produced more value?
The answer may still be AI. But companies shouldn’t assume the answer is AI simply because the technology is impressive.
The goal of measuring productivity is to understand whether an investment is producing a meaningful return. AI should be held to the same standard.
This is where Quality Engineering changes the equation
There’s still an important problem with measuring productivity through feature delivery alone: a feature being delivered doesn’t automatically mean it’s valuable.
It has to work.
This is where Quality Engineering becomes an important part of the AI productivity conversation.
Imagine a team goes from delivering two features per month to four. That’s a great result. But what if defects also increase? What if the team is spending significantly more time on rework? What if QA has to spend twice as long validating AI-generated code? What if more defects are escaping into production? What if developers are moving faster but spending more time fixing what they created?
The organization may have increased its output without increasing its actual value.
That’s why I think the next evolution of AI productivity measurement needs to connect delivery and quality.
For a deeper look at how to rigorously validate AI agents in production, I’d recommend Anantha Subramanya’s article: https://www.ananthasubramanya.com/blog/enterprise-ai-evals
The question isn’t simply, “Did we deliver more?”
It’s, “Did we deliver more valuable, working software without creating a proportional increase in quality problems?”
That’s a much harder metric to game. And arguably, a much better definition of productivity.
From AI activity to business value
I don’t think the answer is to throw away the old metrics. PRs, velocity, development time, token usage, and other activity metrics can still tell you something. They’re useful signals.
The mistake is treating those signals as the final measurement of productivity.
AI is making software easier and faster to produce. That changes what some of our traditional productivity metrics mean. If writing code becomes cheaper, measuring how much code someone writes tells us less. If creating a PR becomes easier, counting PRs tells us less. If AI can generate thousands of lines of code in minutes, code volume becomes an even weaker proxy for value.
The scarce resource starts shifting toward something else: judgment.
Did we build the right thing? Did we build it correctly? Can the business actually use it? Did we reduce the time and cost required to deliver it? Did we create more value than the AI investment cost us? And did we maintain quality while doing it?
That’s where I think AI productivity measurement is headed.
Stop measuring just what AI produces. Start measuring what the business receives.
Because software isn’t productive simply because it was generated.
It’s productive when it works, delivers value, and does so at a cost that makes sense.