Back to Blog
Engineering

When there's more to count, what actually counts?

Story Reviewer  You Make the Call

At a previous company where Mark worked, the team was having trouble with software quality, and the CEO had an idea for fixing it. Why not put out a bug bounty and give a prize to whichever testers reported the most bugs?

Mark wasn't a fan.

I don't want 500 bugs we're never going to fix.

A contest like that rewards the number of reports, whether or not the problems matter. "Everybody can pixel count and find issues and problems," he says, "but are those things we care about? We want quality."

Mark Henzi now leads engineering here at Atono. He got into QA early in his career because, in his words, he really likes to break stuff, and he spent about ten years there before moving into engineering leadership. When we sat down with him recently to talk about how AI has changed his work, the bug bounty came up while we were discussing how to measure AI-assisted engineering.

If your CEO asked you tomorrow how much more your team is getting done with AI, which numbers would you reach for?

What lines of code and pull requests don't tell you

For a lot of teams, the answer is lines of code and pull requests, and AI agents make it possible to produce more of both. In the early days of agentic development, Mark listened to plenty of podcasts where teams talked about pushing tens of thousands of lines of code a week.

That doesn't mean it's better. It just means you're writing a lot of code.

Pull requests have become the popular number to track lately, and they have the same problem. "I can make a one-liner PR," Mark says. "So what's the actual difference?" A pull request might hold a big change or a single line, and either way it counts as one. He has yet to hear anyone describe a metric for this kind of work that feels meaningful to him, "because it's all so contextual."

"It's all the same," Mark says, thinking back to the bug bounty. Like the bug reports, these counts lump very different kinds of work together. So when a count goes up, the more useful question is what changed in the work behind it.

Our own numbers are a good example. Our codebase is twice the size it was a year ago, but a lot of that growth is tests and documentation, so the increase doesn't mean we're delivering twice as much functionality. That's why Mark isn't focused on increasing output for its own sake. His goal is to "let AI augment my developers and help them go faster doing what they do really well, as opposed to just turning out a whole ton of code."

Why small stories make pull request counts more useful

This doesn't mean Mark thinks counts are useless. How a team shapes its work affects what those numbers mean.

His main advice for anyone writing stories for agents is to keep them small, because "agents are much more likely to succeed working on something that's simple, just like a person." He says that's always been true, and it matters even more now. In our sizing sessions, anything bigger than a medium needs a good reason. Otherwise, the conversation turns to how we can break it down. When stories are small, he thinks pull request counts might start to tell you something.

That's a context where number of PRs maybe is a good metric, because if your stories are small, you're keeping your PR count low per story, and you can do lots of stories.

If a story might take one pull request or fifteen, a jump in pull requests could mean many stories got done or just a couple. When stories are consistently small, the count may say more about how many are actually getting finished. He's careful not to overstate it, though, and adds right away that "it all comes down to what actually means something in the environment in which you work."

How we're deciding what to measure

Rather than pick a metric because other teams use it, Mark is taking the same approach he takes with any big problem that doesn't have a strict definition. Start with something small, make it work, build on it, and then optimize. When he looked into what other teams were measuring, he found some focused on lines of code and others on pull requests, tokens, or cost. His response was that we can measure all of it.

So that's what we're doing. Mark asked the team to bring together the data we can readily collect, including tokens and pull requests, so we can see where the opportunities are before deciding which measurements matter most. For him, though, the numbers all point back to the same questions. "It still comes down to velocity," he says. "What are you building and putting out, and where's the value?"

Connecting what we ship to what customers use

That last question is the one Mark is most interested in, and it goes beyond engineering activity.

"One thing I've never really done in my career is take that velocity and translate it into value for customers. What are customers actually using from the stuff you've built?"

Feature engagement data gives us a way to start making that connection. Engagement won't capture everything that makes a feature valuable, but it shows something delivery numbers can't, which is what customers actually do with the work once they have it.

Mark describes the kind of report he'd like to be able to give. "We spent the first two quarters of the year building these features, we had this velocity and this much engagement," he says. "This is the kind of value we actually delivered to our customers over time."

So if your CEO does ask how much more your team is getting done with AI, the numbers are worth bringing. Just bring what's behind them too, from how the work was shaped to what customers did with it once it shipped.

Watch highlights from Daniel's conversation with Mark below.

For more from our team on working with AI agents, read Lex Vorona's take on the decisions your AI agent doesn't ask about.

Stay up to date

Share

Stay up to date

Share