I Put AI Factory Economics on a Meter
In January I argued that the AI factory metaphor was doing more harm than good, and that what a CIO actually needs is a unit of business output with a dollar figure attached to it. That post was reasoning. I hadn’t spent anything to test it.
Then I spent about $121 running a single lab, and the framework’s own warning turned out to be aimed at me.
Pick the unit or stop arguing
The framework’s first demand is the unglamorous one. Before you compare anything, define one unit of business output and denominate every cost in it. For repair work that unit is easy to name and hard to fake: one bug fix that passes a test suite the model didn’t write and never sees.
So I built the lab around 22 certified bug-fix pull requests, each shipping its own tests as the gate. Fourteen arms. Six models, from a Gemma 4 31B on hardware I own up to two generations of Opus, run both inside a constrained repair loop and inside an agentic harness. Every arm either produces verified repairs or it doesn’t count.
That constraint is the whole method. Fix the denominator and a tier argument becomes arithmetic.
The trap I walked into
The framework names token consumption as a failure mode. Read tokens as a business metric and you’ll optimize something that never pays you. I know this because I’m the one who wrote it down, and then I built the first version of this analysis on token efficiency anyway.
Measurement killed it. Three models spanning a fourfold price range consumed within 13 percent of each other. The cheapest of them, run twice against itself, varied 24 percent. The gap between models was smaller than the noise inside one of them, which means every conclusion I’d drawn from token counts was noise with a chart attached.
Tokens tracked nothing. Cost per finished repair tracked everything.
What the unit exposed
Three results fell out once the denominator was right, and I expected none of them.
A paid mid-tier model in the constrained loop scored 16 and 17 across two runs. The free local model in that same loop scored 17 and 17. Buying a better model without changing the apparatus around it bought no separation from free.
Then the apparatus. That same mid-tier model, the one that plateaued at 17 in the loop, cleared all 22 inside an agentic harness for $3.14. Two Opus generations cleared the identical 22 for $13.79 and $12.56. Same finished work, four times the bill.
And the cheapest path was none of those. Let the free local model triage, then escalate only the five repairs the gate hands back. That costs $1.12, against $13.79 for pushing all 22 through a frontier model, for output a validator scores identical.
Before you conclude the answer is just a cheap model, it isn’t. Put that same free Gemma inside the harness and it falls from 17 to 11. The harness multiplies a model that can operate its tools and taxes one that can’t. Instrument literacy is its own capability and it doesn’t track parameter count.
I priced the obvious middle path too, and it lost. Renting GPU to self-host the escalation rung burned $4 to $6 of machine time to solve 9 of 22. Agentic sessions park their context in memory while barely touching the die, so what you’re actually buying is memory per concurrent session. Four entry instances get you about nineteen sessions at $7.44 an hour. The box already on my bench holds fifteen for nothing.
What it cost, including what I wasted
About $121. Call it $73.75 of rented infrastructure and $47 and change in metered tokens.
Sitting inside that total is $5.45 killed by a credit exhaustion I didn’t catch mid-run, and roughly $24 of GPU left idling overnight because I walked away from a meter. I left both in the number. Anyone reproducing this pays for their own failures, and a cost figure that quietly excludes the operator’s mistakes isn’t a measurement. It’s marketing.
This isn’t one lab’s arithmetic, either. An earlier lab priced the same unit on rented Xeon capacity at nine cents a verified fix, six cents at the measured Savings Plan rate, and audited 585 scored attempts to confirm the gate never passed a repair that failed its held-out checks. That last part is what makes the cheap rungs safe to grind on. Not the model. The gate.
So what?
The ladder itself doesn’t transfer, and I want to be plain about that. Mine is conditioned on a specific pool of 22 repairs that a specific free model missed on its first pass, in one Business Process Automation (BPA) workload with a deterministic acceptance test. Yours will look different.
The denominator transfers. Cost per verified unit of business output is the thing you have to define before any of this arithmetic runs, and defining it is what turns a vendor conversation into a measurement. Without it, tier debates are unfalsifiable and everybody in the room gets to be right.
I’d rather be wrong on a meter.
Self-funded. No vendor paid for this work, reviewed it, or saw it before publication. The free local worker is a Google model running on NVIDIA hardware I own, the paid arms were metered OpenAI and Anthropic calls, and the rented legs ran on AWS. Google Cloud is a client of The CTO Advisor LLC. NVIDIA, OpenAI, Anthropic, and AWS are not. The harness is open under the MIT license at github.com/kltownsend/loopcontrolbench. The assessment methodology and the verdict are mine.