An honest look at AI productivity claims, and how to measure 2-3x

A 2-3x productivity claim is meaningful only when it survives scrutiny, and most do not. The gains I stand behind are the ones that show up in metrics tied to delivered work: build success rate, cycle time, code coverage, and reductions in effort and time cost. The claims that evaporate are the ones built on vanity numbers like lines of code or how impressive a single run looked. Here is how to separate the two, with a measured example to ground it.

What does “2-3x productivity” actually mean?

It means the same team delivers two to three times the outcome per unit of effort or time, measured on real work, not a benchmark. The unit matters. “2-3x faster on a toy task” is a ceiling number that tells you almost nothing about a delivery team. “2-3x more shipped per cycle, sustained” is a floor number that tells you almost everything.

The honest version of the claim is always specific: which metric, over what baseline, on what kind of work. A figure without those three things is marketing, not measurement.

Which metrics survive scrutiny?

The trustworthy metrics share a property: they are hard to fake and tied to delivered value.

• Build success rate. Of the work the AI produces, how much actually compiles, passes, and ships. A high rate means the output is trustworthy, not just plentiful.
• Cycle time. How long real work takes end to end. It captures rework, review, and the messy middle, which is where time actually goes.
• Code coverage. A proxy for whether speed came at the cost of safety. Faster with maintained or improved coverage is real. Faster with collapsing coverage is debt.
• Effort and time cost reduction. The bottom line: fewer person-hours for the same delivered outcome.

The vanity metrics are the inverse: lines of code generated, number of prompts run, how loud the demo applause was. They go up easily and predict nothing.

What does a measured 2-3x look like in practice?

A concrete example. On a Singapore Airlines website feature, the team shipped in five weeks what previously took nine. Roughly 95% of the code was AI-generated, build success climbed past 90% within three weeks, and overall effort dropped by more than 60%. Every one of those is a trustworthy metric: time to ship, build success, effort reduction, all on real delivered work, not a demo.

That is what a defensible 2-3x looks like. Not a number on a slide, but a baseline, a measured result, and metrics you would let an auditor see. I cover the methodology in more depth when I speak on this, and it is the backbone of how I run enterprise AI deployment.

The honest takeaway: the gains are real, and they are also routinely overstated. The way to tell the difference is to ask for the baseline, the metric, and the kind of work, then ignore anything that cannot answer all three. Measuring well is also why pilots that look great so often stall before production: they optimised a vanity number and never checked the ones that compound.