Background

As the conversation about the return of investment (ROI) associated with AI intensifies I decide to do some analysis across our Linear database to see if AI has had an impact on engineering at Kerno. Notionally, being part of a small team you can ‘feel’ the productivity gains, but I wanted to anchor this in data.

Preface: During my ROI hunting expedition, I did not link ROI back to customer driven $. This expedition was focused purely on productivity. In next editions I will traverse back to features and $-value creation.

Context

To help you digest this piece, there is context that had a material impact on the numbers:

• Like most companies in 2025 we deemed that AI coding agents weren’t good enough to start writing production acceptable code until around August/September last year.

• The majority of our team were skeptical of LLMs in the application of code generation. This meant that there was an initial education ramp through August and September, where engineers were provided with space to experiment and explore.

• We had 40% attrition during the past 12 months as we changed product direction and some roles were no longer needed.

• We released a major refactor of our product in March which attributed to a spike across all graphs [tickets, tokens, bugs]. While I didn’t adjust for this in the dataset, I will add notes and draw my final conclusion based on this.

• All our engineers have 10+ years of experience.

• While not all code is generated by AI, the rough estimated is that 70% is.

• I have not accounted for the PoC scrapyard or technical debt in this analysis.

Beyond the obvious

If I asked each of my engineers the question… is AI helping you, the answer is an easy yes, but I wanted to look beyond the usage metrics. I wanted to understand the nuances and see if there were correlations across certain dimensions:

Adoption: While every team have AI power users, AI code generation can only have a true org wide uplift if the majority of your engineers have bought in to it and are using effectively*. I wanted to check what adoption looked like across the team.

*effectively means something different for everyone, so let’s just say they are moving beyond basic prompting and investing time to make this tool better for them.

Bug Velocity: Shipping fast means nothing if what you ship is slop, it undermines progress. While software bugs are part of doing business (and always will be), I was interested in seeing if the bug velocity was up or down post introduction of AI.

Tokens: Another trend I was interested in, is how token consumption is trending. My assumption was we would have a sharp rise in tokens as we started using AI initially, but then it would fall somewhat and stabilise as we started using more skills, better scaffolding, memory layers and better prompting.

Summary

The analysis covers 17 months of completed Linear tickets, January 2025 to May 2026, split at the September 2025 adoption point: 2,410 tickets in total, 376 of them bug-labelled.

At a glance:

• +180% uplift in output per FTE

• Ratio of bugs:output is no different between pre and post AI (adjusting for major refactor in March).

• Token consumption starting to stabilise and decline.

Adoption: did the whole team use it?

Total throughput rose by 41% (~117 → ~164 tickets/month), which undersells the attrition story. Accounting for this, per engineer output went from ~9.2 to ~25.7 tickets/month - roughly +180%

The gain shows up across both teams, not in one or two power users: Backend rose +158% (~7.4 → ~19 per engineer) and Frontend +258% (~16 → ~59), though the Frontend figure is volatile on a small team and is best read as directional. The signal is a broad, sustained uplift.

Share AI Builder Series

Bug velocity: quality or slop?

No defect explosion, and no miracle either. Bugs were already being closed at ~17–31/month before AI; afterward they rose a modest +52%, in line with simply doing more work. Bug share held steady in a 12–27% band the whole period, and bug cycle time actually fell (0.54 → 0.19 days). This is a nice win!

Adjusting for March major refactor (non-normal), the average ratio would restore to somewhere between 15% - 17%, which is lower than pre-AI era.

Tokens: did spend stabilise?

Based on raw data — No. The expectation was a spike followed by a plateau as scaffolding improved. Instead, once all tooling sources are counted (the first pass undercounted by ~44%), tokens per ticket rose from ~10M (Nov–Jan) to ~16M (Feb–May), and total monthly spend climbed to a ~4.2B peak in March. Productivity went up, but cost per unit of work went up with it — the opposite of the efficiency plateau the scaffolding thesis predicts. The one caveat: spend dipped in the last two months (4.2B → 2.3B), which might be an early peak, but two points isn’t a trend yet.

But this does not adjust for the March refactor. The MoM change tells a different story, where in % change, I am seeing a decline in April and May, which is the payback from the continued investment in the underlying infrastructure and tooling making AI better. If June also provides a down month, it could be fair to conclude that token spend is starting to stabilise as we are getting better at reusing context and knowledge.

The team has deployed all the modern infrastructure to make AI work reliably - skills, scaffolding, memory, shared context across repo, gaurdrails and we also have started to use our own Kerno Intelligence Tooling (KIT)* since May.

*KIT is a set of code intelligence tools delivered via MCP that replaces broad, expensive searches with precise, indexed lookups. Instead of reading entire files, agents can jump directly to symbol definitions, find all usages across a codebase, trace call chains, and list endpoints without touching unnecessary code, which can reduce token consumption by 90%+.

Supporting signal: cycle time

Median time-to-close fell 60% (1.79 → 0.72 days, trending toward ~0.1 by spring 2026). Work isn’t just higher in volume, it’s closing roughly twice as fast, which corroborates the per-person productivity story.

Again, bug cycle time actually fell by 65% which was a pleasant suprise.

Conclusion

Overall AI has accelerated productivity across our engineering organisation. The ROI is clear. Without AI, we would have not been able to ship features at a record pace while absorbing losses in our team.

While this is giving me signal, what this report doesn’t capture is the significant amount of investment we have made to the underlying infrastructure to make AI code generation work for the team.

Skills, memory layers, benchmarking, documentation, prompt-engineering, knowledge sharing, testing and much more have been a significant upfront and continued investment to make this tool a reliable…..tool. This is the sunk cost of using AI. I estimate this for our team size to be equivalent to 0.5xFTE* per year ~$50,000.

This translates to 33% of the total cost of AI enablement ownership (tokens + cloud + engineering).

For every engineer post AI, the output is 3x, for simple maths, assume salary is $100K, which means, to achieve the same output, you’d need 3 engineers at $100K each.

So the ROI math for a 7 person engineering team is

With AI: $150K + $700K = $0.8M

Without AI: 21 x $100K = $2.1M

NET ROI = $1.3M

While the eagle eyed data scientist in you might be quick to point out the flaws - this ROI is not tied to realised gains, aka impacting the P&L in the form of tangible $$ (not making that public) or cost cutting. But for me, it’s enough to give me directional signal to continue the investment, and even ramp up the investment. AI has been additive.

Translating this to bigger more complex organisations needs to factor in Conways law. I have found that scaling anything beyond two pizza teams requires a totally different implementation plan and timeline expectations. I also don’t think the AI enablement cost scales linearly, it’s more logarithmic.

Actionable takeaways

• Assessment of skills we use, and upgrade accordingly

• Weave in PR analysis to help understand where bottlenecks are in cycle time.

There’s a quiet crisis spreading through the founder community right now. Someone posts a video of Cursor or Claude spinning up a product or feature in twenty minutes that took your team six months to build. Your Slack goes quiet. Someone links it in the engineering channel with no comment. You close your laptop, stare at the ceiling, and start doing the maths on runway.

Then the pile-on begins. VCs, Podcasters, Commentators, DevRels start chirping:

Let me be direct about what’s actually happening: these people are not building products. They are building audiences and your existential crisis is the content. The demo is not the point. Your self-doubt is the point. Engagement needs fuel, and founder anxiety is a great source.

We are in an era of Sensflationism.

Sensflationism is the word I am coining to describe the emergent dynamic where sensationalism and conflation feed off each other until prototypes pass as products and hearsay passes as the reality.

It works like this: AI genuinely enables people to build faster. But “faster” gets conflated with “complete,” “complete” gets conflated with “production-grade,” and “production-grade” gets conflated with “indistinguishable from a product built by people who’ve spent years inside the problem.” Each step seems reasonable. Together they collapse the gap between a two-hour prototype and a two-year product into a single confident post on X…. add the YC logo, and they are destined for the moon.

The people amplifying this usually don’t have the expertise or time to build any sort of depth into the demo. AI compresses execution time but it does not compress the judgment that comes from watching your product fail in production, from understanding the edges of a domain, knowing where the hard problems actually live, not inferring. That knowledge accrues slowly. It always has. The models got faster. Understanding the problem and building the real solution didn’t.

When these fly by demos go viral, VCs start bringing it up in conversations and sales team start losing confidence in the defensibility of the product. Suddenly for no valid reason, you’re facing a crisis of confidence.

I got hit by it. 11pm on a Tuesday. A demo of a skill that looked at first glance like it was doing exactly what we’ve spent years building. I felt the dread that every founder knows and nobody talks about.

But I didn’t dismiss it, instead I went and tested it myself.

What I found didn’t just put my mind at ease. It reassured me on how I think about what makes an AI product real and valuable, and our business - a business.

The Mistake Everyone Is Making

Before I get to the experiment, I want to name the misconception clearly, because it’s underneath almost every conversation I have with founders right now.

People are confusing the prototype for the product. They’re not the same thing. They’ve never been the same thing.

The model is one component. It is a powerful one, getting more powerful every month. But a model alone doesn’t solve a problem for a customer any more than an engine alone gets you from A to B. What gets you from A to B is the complete system — the chassis, the fuel system, the steering, the brakes, the decades of accumulated engineering knowledge about how to make all of those things work together reliably at speed, in bad weather, with a distracted driver, and then you need to figure out how to reliably mass produce them. Take the engine out of that system and show it to someone on a workbench. It’s impressive. It’s not the whole car.

The Founders panicking right now are looking at a very impressive engine, connected to the chassis and a fuel-system and forgetting everything else.

What You’re Actually Selling

An engineer I had lunch with last week said something that forced some deep thinking..

He didn’t mean trust in a brand-values, mission-statement sense. He meant it technically. Trust that the product does what it says it does, ninety-nine percent of the time. Trust that the creators have seen enough failure modes to handle the ones you haven’t imagined yet. Trust that when something breaks at 2am, the system fails in a way you can reason about, not in a way that silently corrupts your data and moves on.

That’s the real product. Not the feature, not the model, not the demo. The accumulated engineering judgment about how to make something work reliably for real users in real conditions.

This is harder to see than it sounds, because the demos are genuinely impressive. The prototype works. The AI reasons about the problem correctly. The output looks right. And so the natural conclusion is: the hard part is done. What’s left is just polish.

The Experiment

Kerno is a validation engine that helps engineers (and their AI) catch breaking changes in real-time. It does this by running just-in-time integration tests against your full stack as you code.

What makes it interesting isn’t that it writes tests. It’s the scope of what it manages. It inspects your codebase, understands your service architecture, builds the environment if one doesn’t exist, writes tests suited to your actual architecture, and crucially, maintains that test suite as your code evolves. When you change something, Kerno doesn’t just rerun old tests. It understands what changed and updates the suite accordingly.

That last part is the hard part. That’s the product.

I wanted to test a hypothesis: could I replicate this with a Cursor skill driven agent that hooks into my IDE and runs on every code change? If the answer was yes, that would be genuinely alarming.

So I spent an afternoon building it properly. The skill was designed to do the following on any code change: detect the affected service and understand its shape; spin up the relevant environment; generate and run integration tests against the change; show a clear before/after view so I could see exactly what the change did; and update the test suite to reflect the new expected state.

On paper, this sounds tractable. These are all things an LLM can reason about. And in a demo, it looked like it was working.

The tests ran. All bar one passed. The output looked clean. If I’d glanced at it and moved on, I would have shipped.

╔═════════════════════════════════════════════════════════════════
║ INTEGRATION TEST RESULTS — POST /api/orders                    ║
╠═══╦══════════════════════════════════════════╦══════════╦══════╣
║ # ║ Test Case                                ║ Status   ║ Time ║
║ 1 ║ Happy path — valid order, payment OK     ║ ✅ PASS  ║ 312ms║
║ 2 ║ Missing required field (productId)       ║ ✅ PASS  ║  48ms║
║ 3 ║ Product not found (invalid ID)           ║ ✅ PASS  ║  61ms║
║ 4 ║ Insufficient stock                       ║ ✅ PASS  ║  73ms║
║ 5 ║ Unauthenticated request (no token)       ║ ✅ PASS  ║  22ms║
║ 6 ║ Wrong role (admin token)                 ║ ✅ PASS  ║  19ms║
║ 7 ║ Duplicate order within 30s               ║ ✅ PASS  ║  88ms║
║ 8 ║ DB write fails mid-transaction           ║ ⚠  SKIP  ║    — ║
║ 9 ║ Response shape matches OpenAPI contract  ║ ✅ PASS  ║ 140ms║
╚═══╩══════════════════════════════════════════╩══════════╩══════╝

But when I dug into the docker logs, the database had never actually started. The tests were running against nothing, succeeding because there was nothing to fail against. Every green checkmark was a false positive.

To make it worse: one of those tests should have failed. I had deliberately introduced a breaking change to validate the setup. The skill didn’t catch it. It didn’t flag it. It just reported success.

From start to finish, the experiment took three hours for a single endpoint change. My honest estimate for extending this across a real codebase: at least ten hours of monthly maintenance overhead, not counting the cost of acting on incorrect results.

When someone tells you they built your product in two hours, they built the part that takes two hours.

Share

Why This Will Always Be Hard

The skill didn’t fail because the LLM reasoned badly. It failed because the problem is harder than any prompt can capture in three specific ways that apply to every AI product, not just this one.

Illusion of Success

AI systems don’t know when the environment can’t be trusted.

The model reasons about the happy path. It doesn’t natively detect when the ground beneath it has shifted. In my experiment, the database never started. The tests ran anyway. They all passed. A system that crashes tells you it broke. A system that silently succeeds against nothing trains you to trust it. Every time you glance at green output and move on, false confidence compounds. And then something real breaks in production and you have no idea why your safety net didn’t catch it, because as far as you knew, the safety net was working.

This is the dangerous failure mode: an AI can mimic the output of a reliable tool without replicating the properties that make it reliable. In a demo, those two things look identical. In production, they’re completely different.

Handling this means encoding defensive logic that has nothing to do with the AI: detecting when conditions aren’t met, failing loudly rather than silently, refusing to produce output that would be worse than no output at all. That only gets designed in by someone who has watched the system break enough times to anticipate it.

Engineering judgement is a moat

AI outputs require interpretation that the model can’t provide.

The before/after comparison — the thing that would have made this actually useful — was absent from my experiment. No meaningful diff. No signal about what changed in behaviour. Just a wall of passing tests and the illusion of confidence.

This is the general problem: a result that looks right may not be meaningful. A result that looks wrong may be a fluke. The hard work in any AI product is building the layer that knows the difference — what constitutes a genuine problem versus an artifact of the environment, what confidence threshold should trigger action versus caution, when to surface something to the user and when to stay quiet. These aren’t model decisions. They’re product decisions, shaped by watching real users misread real outputs and building guardrails around the patterns you observe.

Consistent Accuracy

Coherence over time is an architecture problem, not a model problem.

A prompt is stateless. Your product isn’t. From start to finish, my experiment took three hours for a single endpoint change, and that’s before the maintenance burden compounds across a real codebase. The longer a customer uses your product, the more context has accumulated that a one-shot AI call can’t hold: the history of what’s changed, the preferences that have emerged, the failure modes that have been resolved. Products that earn long-term trust behave consistently across time, across team members, across the entropy that naturally accumulates in any real-world system. That consistency doesn’t come from the model. It comes from the user feedback, varied customer base and the subsequent architecture built around it.

The moat no one talks about

Hard won tribal knowledge

There’s a layer beneath all three of these that is not discussed enough: LLMs are trained on what has been written down. And most of the pain your product solves has never been written down.

The really valuable domain knowledge doesn’t live in documentation, blog posts, or Stack Overflow threads. It lives in the conversations you’ve had with frustrated users at 4pm on a Friday. It lives in the support tickets that revealed how people actually misuse a tool versus how they’re supposed to use it. It lives in the three-hour onboarding call where a customer walked you through their environment and you realised your mental model of the problem was wrong in a specific, important way.

That knowledge isn’t in any training set. It can’t be prompted out of a model, because it was never written down in the first place.

This is the part of domain expertise that compounds invisibly. Every customer conversation, every edge case, every “that’s weird, why does it do that” moment adds to a map of the problem space that no language model has access to. The model knows what everyone knows. You know what nobody wrote down. That asymmetry is durable in a way that technical implementation details often aren’t.

What This Means For How You Build

The model call is maybe twenty percent of the surface area of a working AI product. The other eighty percent is everything that makes the output trustworthy: the defensive logic, the customer discovery, the coherence over time, the UX that helps a user understand what happened and why.

The LLM is the engine. But what you’ve built, the orchestration, the failure handling, the domain knowledge , the coherence that holds up at 2am when something goes wrong, that’s the rest of the car.

So ask yourself: is your product hard because of the model, or because of everything around the model?

If it’s the former, you have a real problem, not because AI is going to replace you, but because you haven’t built the thing that’s hard to replace…yet.

If it’s the latter, if the hard part of your product is everything that surrounds the model, then you’re probably safer than the demos suggest. But you need to know why you’re safer, specifically enough to explain it. To investors who ask. To engineers who wonder. To yourself at 11pm so you can cut through the sensflationism.

The engine is not the car. You built the car. Now make sure you can articulate it.

Introduction to the Halting Problem in Mathematics

There is a short list of ideas from early computer science that are (a) worth knowing and (b) just plain fun to think about. One of these is the Halting Problem. This is a landmark undecidability problem from computability theory - the field that describes what can and cannot be computed.

The problem centres around the question of whether or not any given program will ever stop running, or just get stuck in an infinite loop (assuming it is allowed to run forever by the hardware). The existence of an algorithm that could decide this for all programs is a proven impossibility. Whether a program will stop or not is mathematically undecidable. This is the most well known and only undecidability problem in computing. It belongs to a branch of mathematical results developed by luminaries of early 20th Century mathematics and computing including Hilbert, Gödel, Church and Turing, and culminated in Rice’s theorem, proved in 1950. Rice’s theorem generalises the halting problem statement to say that all non-trivial semantic properties of programs are undecidable. Here, a non-trivial semantic property is a behaviour that some, but not all programs can have. For example, it is not possible to determine in general whether a program will produce a specific output for a given input or whether a program computes a particular function.

Why This Matters for AI-Assisted Software Development

This has some important consequences for software development and testing in the age of generative AI and coding agents (AI-DLC). Static analysis of code can already be done with linters, type checkers, model checkers and formal verification (like TLA+), and these are very powerful tools. We would obviously want to enhance these methods by adopting new generative AI technologies. However, Rice’s theorem means that, impressive as they are, static analysis of code by LLMs, as with any other method, will always be unable to guarantee completeness of testing. Even if we reach the mythical AGI, no LLM can predict the behaviour of all or any given program just from reading their code, and “any given” may include yours.

Take halting, for example. Halting, within a given amount of time, is an important behaviour of code, especially in a world where computation is a physical process that has to be bought and paid for, and users usually want outputs. To determine this, eventually the agent is going to just have to execute the code (note that I say “given amount of time” - as this is all that the simulation can tell us too).

This is a fundamental limit on AI testing even before we introduce anything complicated like distributed systems into the mix, which can add more undecidability problems.

The Case for Execution-Based Testing

One way to overcome this is to build an environment where the code can be executed, and then use an agent to define the input and expected output. This is already done using hand-written CI/CD processes and integration tests, but these are very time intensive to write and maintain, especially when a codebase is changing quickly. As we adopt agents to write more code, they will change ever more quickly. AI agents have developed to the point where they can autonomously write the correct configs to start a virtual machine, and repeatedly run code on the virtual machine. Where we have distributed systems, this can be done for each machine in the network, and we can run the whole system in order to probe behaviour, such as race conditions or cascading failures (though non-deterministic behaviours may require repeated runs or chaos-engineering techniques to find). Likewise for systems made up of multiple applications.

The Challenge of Distributed Systems

Monitoring and analysing the behaviour of such a distributed system is inherently difficult, hence the existence of the multibillion dollar observability market. Anticipating what effect any one change in the code will have across the system is difficult for engineers. Leslie Lamport once defined a distributed system as “a system in which your computer can be taken down by a computer that you did not know even existed”. This problem is no less difficult for even advanced AI agents. All models have a limited context window that narrows their view of a system. This limitation precludes them from exploring and anticipating very many possible behaviours of a complex software system, and the undecidability problems discussed above reinforce this. Further, when working with agents to write code, and to do so with increasing volume and speed, that option of the engineer understanding the code to a level at which they can anticipate how any one change will affect the whole system is becoming increasingly untenable.

How Running The Code Solves This

At Kerno, we believe that anyone writing production software with AI assistance, especially in distributed systems, will need to do execution-based testing. Kerno can take a repository, index the code, and then write integration tests for any endpoint in the system. It can then orchestrate and run any of the services needed to perform these tests by executing the code. It can do all of this seamlessly with agents. Further, it can monitor any changes made to the codebase, adapt the tests to the code, and give the user a clear measure of how a change to the code will change the behaviour of the system. Execution-based testing is not a panacea for software development. It cannot give the user certainty, but it can provide empirical evidence for behaviours, and time bounds for execution of any steps. Kerno can make this process markedly easier.

Disclaimer - this is not another McKinsey or Gartner enterprise playbook dressed up for startups. Instead, findings are grounded in conversations with more than fifty engineering leaders and staff engineers across Seed to Series D companies in Europe and the US. Collectively, these teams have raised over $1B+ and are building some of the fastest-moving AI companies today. The patterns here are not aspirational, they are already happening under real delivery pressure.

As a startup it can be hard to know if you are doing things as fast as your peers, adopting the right technologies, staying true to the agility required to navigate product market fit. Most literature about the AI-DLC is written for enterprises, 1000+ employees, who are bound by regulation, Conway’s law and many more constraints that make them akin to a cruise liner versus a speed boat!

In startup land, when people talk about “AI adoption,” they often mislabel it as ‘vibe coding’, which frankly is f***ing infuriating. In practice, not everything is vibe coded, the real answer is more nuisance and really belongs on a scale to understand: how deeply AI is embedded across the development lifecycle and how much autonomy teams are willing to give it.

I find the model Steve Yegge uses in his ‘Welcome to Gas Town” article a good mental model to help outline where AI adoption is in terms of autonomy and trust.

The table below outlines each stage and creates a heatmap of adoption:

At the lowest end of the spectrum are teams that occasionally use code completions or ask chat-based questions. Among high-growth startups, this mode barely exists anymore. Most teams have moved to IDE-based agents that can write code and run tools… but only with explicit permission. This isn’t a technical limitation - it’s a psychological one. The tooling is capable of more. Teams simply don’t trust it yet or have invested in the proper guardrails (or are moving so fast that guardrails aren’t fully baked yet).

As trust increases, permissions are relaxed. Agents act without constant confirmation. The IDE stops being a place where code is written and becomes a place where diffs are reviewed. Some teams push further, running agents directly from the CLI, letting changes stream past faster than any human could realistically inspect. A very small number go even further, coordinating multiple agents or building their own orchestration layers.

Most teams stall before reaching that frontier.

What’s notable is that this stalling doesn’t happen because AI makes too many mistakes. It happens because teams cannot prove that the AI is correct.

Despite wide variation in tooling and maturity, these companies share the same priorities.

Shipping speed comes first.

Quality is not enforced through rigid upfront processes, but through fast feedback loops. AI dramatically accelerates these loops - up until the point where humans must intervene to validate outcomes. A lot of engineering managers we speak to share the same philosophy: fast fail, rapid feedback loops, regenerate is a hallmark of reliability.

Everyone-as-a-‘coder’

These companies also share another trend - more non-engineers are empowered to ship features.

To manage risk, teams implicitly define trust boundaries. Customer-impacting issues remain human-led. Core backend logic is increasingly AI-generated, but reviewed carefully. Frontend and UX work is mostly delegated to agents, with humans approving results. Prototypes and internal tools are fully automated. The approach is rooted in pragmatism. The more visible the blast radius, the more validation is required.

Intent driven development

In the Enterprise, we are seeing the rise Spec-driven development, (the reignition of) test-driven development, and behavior-driven development, but these have not trickled down to startups. Instead, startup teams rely on authoritative context: Combination of PRDs, tickets, designs, documentation, and sometimes tests, all bundled together to express intent and form the origination prompt. The format matters less than the clarity of the outcome.

Startup AI-DLC

All 50 startups seem to converge around the same process.:

• More emphasis on work up front to provide appropriate context and instructions. Using LLMs to help consolidate and organise the context into a single prompt

• Then using their AI-IDE or Claude, to carry out the planning and reasoning, with engineer-in-the-loop.

• The next step is a new step in the process - validation step. Here the engineer tests the change locally, and tries to understand what new code/behaviours the AI has introduced.

• Then normal service resumes with CI, with some companies opting to use PR splitters to breakdown larger PRs into more digestible chunks (success varies).

Developers now do most of their work and discussion before code is written. They clarify intent, debate tradeoffs, and ensure the problem is well-defined. That context is handed to a reasoning model for planning, and then to agents for execution. Tooling handles scaffolding, security, and test generation. Humans return at the end to review.

But, bottlenecks exist. Truth be told, code was never the problem for a good engineer - it was the layers of scaffolding around it.

Code review has become the dominant bottleneck. Pull requests are larger. Diffs are harder to reason about. Developers trust code written by other humans more than code written by agents, even when the agent followed instructions perfectly. When intended behavior and actual behavior diverge, the only way to detect it is through manual validation—reading code, replaying scenarios, or waiting for issues to surface in production.

At this point, AI has already moved the bottleneck. Writing code is no longer the slowest part of software development. Understanding whether the code does the right thing is.

Emergence of the Continuous Validation Phase

This is why validation—not generation—is becoming a critical investment for engineering leaders to fully enable the AI development lifecycle.

Validation acts as the bridge between what was intended and what AI has generated. It is how engineer leaders turn autonomy into trust. Without it, every increase in agent capability simply shifts more cognitive load onto human reviewers. Faster code generation produces larger diffs. Better planning produces more complex systems. And without a reliable way to validate behavior against intent, humans are forced to slow everything down. And this is happening before PR, giving engineers faster signals closer to the point of generation.

The frontier teams are not winning because they have better models. They are winning because they are systematically reducing the cost of validation by creating faster feedback loops. They are finding ways to answer a simple question quickly and confidently: ‍

does this behave the way we expect, in production-like conditions?

The next leap forward will not come from agents that write more code. It will come from systems that can validate behavior, surface mismatches, and close the feedback loop before humans become the bottleneck again.

That is the opportunity that these companies are addressing. Closing this gap means faster shipping, better execution and more efficient AI-DLC. Speed of execution directly correlates with product market fit, so payoff is there.

Next bet: Claude as the SDK for the AI-DLC

In the last two weeks, I have had conversations with two very experienced technical serial founders. Both have fully moved away from Cursor, and are all in on Claude. Not just to generate code, but to orchestrate their local stack. Claude skills and hooks, call the appropriate tooling, and developers step when Claude is unsure or when it’s a critical change and requires more human-in-the-loop.

So, while the Gas Town model shown at the beginning of the article shows that there is a lot of trust to be built and diffusion before we get to agentic orchestration, my bet is that startups using Claude (or similar) will get there faster, within the next 6 months.

Introduction: Why Bubble Talk Matters—and Doesn’t

Much of the attention of the financial and tech press for the past few weeks has been on the subject of a possible AI bubble, not unlike the Dotcom bubble, when that might pop, and what that might look like. See for example the charts below from Bloomberg (https://www.bloomberg.com/opinion/newsletters/2025-10-01/assume-an-ai-bubble-what-difference-would-it-make). From the point of view of an AI builder, I’m going to outline here why some of the things the financial and tech press discuss with regards to the bubble do not necessarily matter to us, and what we’re thinking about instead. To frame this, we note that a sensible strategy for any company is to build a product that people want to pay for, and deliver it at a price that allows you to make a profit. We may even look at profit as a constraint rather than a goal (a point Rory Sutherland from the Spectator has made).

Why an AI Bubble Is Largely Immaterial to Builders

In some ways, a bubble in the industry is immaterial to this. A bubble is about public and private investments that can’t be recouped from later cash flows. Analysts have become particularly worried about this bubble because the spending involved is now responsible for much of the growth in the US economy, and an end to it could bring about an uncomfortable recession. Some economists argue that the sheer size of the bubble would cause problems that would propagate through the economy if it pops or deflates. Others have argued that this is less likely because the AI bubble is being financed more by current revenue of the large tech companies than the massive debt involved in past bubbles. There is also just a lot of wealth searching for a place to be deployed well (see buybacks, growth in M2 money supply, large cash positions of big names ranging from Apple or Berkshire Hathaway).

These things don’t concern builders per se. When it comes to AI products, bubble or not, people will still use any product that they find value in. If the bubble pops, fundraising will still be possible because of the sheer amount of un-deployed wealth mentioned earlier looking for a return. The pain will be felt in share price and ability for companies to borrow on open markets. If you have a real product that can viably make money, you’ll be fine.

Framing the Real Problem: Value vs. Cost

More easily said than done, however. Making a viable product depends on (A) how much people will pay for it and (B) how much it will cost to provide that product. (A) is really dependent on a lot of factors that re not entirely relevant to the bubble talk. It really depends on how much value your product provides, market size, and how tight economic conditions are. (B) is very dependent on the bubble, because the cost of tokens and inference matters a lot, and there are several interconnected factors across both the finance and the technology elements of the bubble that influence this.

1. Token Usage Is Being Massively Subsidised

Probably most importantly - everyone’s token usage is being massively subsidised by the foundation model companies. From some excellent reporting from Ed Zitron, the model builders are losing a lot of money by undercharging for their products. Worse still, OpenAI’s own cost of inference is being subsidised by Microsoft, as they are charged between 1 and 3 times less for compute than other enterprise customers. This can’t continue, and some day soon we may well have to pay much more for the use of any AI tools based on GPUs.

2. Scaling Has Ended

Scaling has ended (which any physicist would have told you it always would). See the ARC AGI leaderboard (https://arcprize.org/leaderboard). It looks very much like logistic growth of capability vs. cost - and that’s with cost on a log scale. Instead, improvements are being made through the use of reasoning models - running jobs through a model multiple times. This is very expensive.

3. Efficiency Gains Through Distillation and Smaller Models

In contrast, there are efficiencies to be gained either by model distillation (making the models smaller). Many parameters (think “parts”) of frontier LLMs are redundant, but you can’t know this when initially training them. Large efficiency gains can be made by systematically removing these. You can also use lower precision in the calculations of the models. Deepseek does this. It can also lead to improvements in other types of software, like climate models. Or you can use smaller models, trained for specific tasks. We think the field will look much more like this in a few years time. At present, OpenAI says that some of its larger models use a router model to choose which of a list of smaller models to use for a task. However, it is difficult to predict what the performance and cost of models will look like in the coming years. Just two years ago, some in the field were convinced that scaling would continue forever.

4. LLM-Based Software Does Not Scale Like Traditional Software

LLM based software does not scale in the same way as other traditional software does. For a database technology, or a website, or anything hosted on a user’s machine (like anything from Adobe, Microsoft Office or Mathworks) the marginal cost of new users is either near zero, or scales sub-linearly. For LLMs, this is linear. Every new user costs you money.

5. Physical Infrastructure and Energy Constraints

Bookending this list is another very important factor - the constraints of physical infrastructure and especially energy. This is the point that the press has mostly focussed on in the past few weeks. There is clearly a difficulty in building enough physical infrastructure to satisfy users’ demand (or predicted demand) for AI products. Any of the factors in point (3) may not help with this due to the Jevons paradox - improvements of efficiency in a technology might not decrease resource usage due to increased demand brought about by lower costs. An example of this is shown in the chart from Menlo ventures (https://menlovc.com/perspective/2025-mid-year-llm-market-update/), which shows that spending by enterprises has not necessarily slowed as models have become more efficient, they just use bigger models.

The resource causing the tightest constraint is quickly becoming electricity. Data centres accounted for 1% of electricity usage in the US at the start of the decade (order of magnitude). It is projected to be 10% by the end of the decade. Adding capacity to the grid can be difficult (for reasons we don’t nearly have time to go into), but suffice to say that this may well become enough of an issue that in the future tokens could well be priced based on the marginal cost of electricity alone. I think this would have profound consequences for the industry and how it interacts with society.

Conclusion: What Builders Should Really Care About

This set of interrelated issues and questions is what AI companies should really be worried about when we talk about a bubble - what will the performance, but especially the cost of models look like going forward. Individuals who like to use AI tools (quite a lot of us, even if people won’t admit it) should also think about this. Our AI usage is being heavily subsidised and if the bubble pops, this is going to end. When building AI tools we really need to think about how much a token is going to cost in the future and how wells we can use it.