AI Builder Series
AI Builder Series
Measuring What Actually Matters: AI Developer Velocity and Cost of Getting it Wrong
0:00
-51:46

Measuring What Actually Matters: AI Developer Velocity and Cost of Getting it Wrong

Featuring Max Kanat-Alexander, Executive Distinguished Engineer at Capital One

Max’s Linkedin

Max has spent his career trying to understand how developers actually work — at Google, LinkedIn, and now Capital One, where he helps to set and drive the strategy for DevX enabling 14,000+ Capital One technologists to be efficient, effective, and happy with software development.

Podcast Timeline:
01:50 — Foundations of measurement: goals, proxies, and perverse incentives
10:54 — Measuring developer productivity in the AI era
14:47 — From power users to the whole team
23:07 — Building awareness, understanding, and a learning culture
29:10 — Advice for engineering managers and leaders
31:48 — The shifting pipeline: DevEx fundamentals and feedback loops
39:08 — The future: process, judgment, and expertise
48:42 — Closing thoughts: quality over velocity, and sign-off

Takeaways from a conversation with Max on the AI Builder Series

1. Start with the goal, not the metric. The single biggest mistake teams make is grabbing off-the-shelf productivity metrics and bolting them onto a business they haven’t deeply understood. Every business is in a different state with different problems. “PRs per engineer” isn’t useless and it isn’t the answer — it’s one signal among many. The job is to triangulate from multiple imperfect proxies toward the truth.

2. Metrics improve developer experience. They don’t manage people. The moment a metric becomes a performance lever, it stops being useful for anything else. Engineers will split PRs into nonsense, spin up background jobs to burn tokens, and game whatever you point at them. You lose the diagnostic value of the signal and create perverse incentives in one move.

3. AI hasn’t changed what you care about. Only how you get there. The outcomes still matter: quality, reliability, customer impact. What’s changed is the process by which software gets built. So stop trying to correlate token usage to gross revenue. A business is a machine that produces results, and the machine matters.

4. Expect a dip before the lift. Every credible study shows productivity drops when teams first adopt agentic development. Leaders need to say, explicitly and out loud, “this is a time to learn and it’s okay to spend extra hours messing around with the agent.” Implication isn’t enough. ICs need permission.

5. Get resistant engineers over the line with a magical experience. Max’s go-to workshop: hand the team an incomplete spec for Rogue (the old text-based RPG), let them build it with Claude Code or Codex, then show off the results. The point isn’t to teach code review, it’s to recreate the feeling every engineer once had when they first told a computer to do something and it did it. That feeling is ten times stronger with agents. People want it back.

6. The fundamentals still win. The teams seeing the biggest lift from AI aren’t the ones with the fanciest agent rollout. They’re the ones who already invested in great testing, strong static analysis, automated guardraill, the unglamorous developer-experience basics. Agents amplify what’s already there.

7. Move feedback onto the developer’s machine. If validation only happens in a fifteen-minute CI run, agentic development doesn’t work. Max optimized his personal test suite from 30 seconds to 11 seconds because the agent runs it so often the difference is enormous. If you only do one thing to enable agentic software development, do this.

Share

8. Devs only code about 20% of the time. Go find the other 80%. Don’t solve it by mandating more coding. Investigate where the time actually goes, like outdated processes, unclear requirements, approval bottlenecks, product-side confusion. That’s where the real acceleration is hiding.

9. Where humans still matter: process, judgment, expertise. Agents are great at “I know the result I want, deliver it.” They’re bad at “here’s a process to build something nobody has seen before.” Scott Hanselman’s observation that Max quoted: people who clone Minecraft with an agent are letting the word Minecraft do all the heavy lifting. The further you get from “duplicate something that already exists,” the more your judgment matters.

10. The most underrated story right now: quality, not velocity. Everyone’s talking about how much faster software gets built with agents. Max thinks the real story is how much better it can get built. Writing great tests is easier than ever. Refactoring is easier than ever. Throwing away an hour of work and starting over is easier than ever. The opportunity isn’t quantity, it’s a level of craft we couldn’t reach before.


“Every time I have abandoned my judgment and expertise to the agent, I have only gotten myself into trouble. What I have yet to hear from anyone is: I looked at the code and decided I didn’t need to look at it.” — Max

Sponsored by Kerno

Discussion about this episode

User's avatar

Ready for more?