If you can’t get GPUs, can you use CPUs?
For one class of work, yes. Not for production serving. Not for latency-sensitive work with a person waiting on the other end. But for batch work with a checkable answer, a CPU you have already paid for will do verified work at a cost that is hard to argue with, and it will tell you honestly when it has failed.
I know that because I could not get a GPU and had to find out.
Sixty-three verified bug fixes in two and a quarter hours on a rented Granite Rapids instance in AWS. About nine cents of on-demand compute per verified fix. Across five hundred and eighty-five scored attempts spanning three models, four precisions and two substrates, not one failure shipped as a false success. The work that cleared was verified work. The work that failed went to a human queue.
That is the answer. The rest of this is why the question came up, what the bench actually ran, and where the answer stops.
Why the question came up
The question I started with was not about CPUs at all. I had a fine-tuned model and I wanted to know whether hosting it myself beat the managed alternatives in a model garden. That is a question a lot of enterprises are asking right now, and it is not answerable from documentation. It needs a bench.
So I filed a self-serve quota request for four L4 GPUs on a project with a clean history and a payment method on file. The request was created and denied inside two and a half seconds. I filed again at half the ask, on the theory that the automation was threshold-sensitive, because it had been threshold-sensitive on the CPU side that same morning. Denied in two seconds again.
The easy read on that is that automation says no because automation says no, and a serious buyer picks up the phone. I picked up the phone. I went through the sales channels the way an enterprise with a budget and a use case is supposed to. That was over two weeks ago. I have not heard back. A committed H100 request on the same project sat unresolved for fourteen days and counting while all of this was going on.
So the accelerator lane, for an account of my profile, answers in two seconds when the answer is no and does not answer at all when the answer might be yes. That is not a quota story. That is what the market looks like from underneath.
The CPU lane behaved like a different material entirely. It granted a family quota bump in six minutes. It stocked out in one region and opened on the first try in another. It negotiated. Once I stopped treating it as a consolation prize, I found it had a job.
What the accelerator vendors are arguing
NVIDIA has been making a specific and correct observation about agentic workloads: the loop is not a single generation pass. The model thinks, calls a tool, waits, thinks again, calls another tool, waits again. The best public measurement of this comes from TraceLab, a University of Washington trace of roughly 4,300 real coding-agent sessions from 43 developers over eight months, spanning Claude Code and Codex across more than twenty model versions: about 357,000 LLM steps and 432,000 tool calls of actual day-to-day use, not benchmark replay. Their finding: within a request, tool execution is 59.8% of response time against 41.0% for generation. An average request takes 4.3 minutes end to end, and the loop is mostly self-driving, with the agent averaging about eight LLM calls and eleven tool calls to resolve one request. During the majority of that time, the accelerator is holding weights and KV cache and waiting.
The conclusion NVIDIA draws is coupling. Put the CPU close enough to the accelerator that the handoffs stop costing anything, share a coherent memory space, elevate the CPU from a supporting role to a co-equal part of the machine. Their newer parts widen the coherent link between the CPU and the GPU and are positioned explicitly for agentic work. Intel makes a version of the same argument from the other direction: if tool execution is most of your wall-clock time, the industry-standard eight-GPU server with its CPU sized as plumbing is provisioned wrong, and you should buy more CPU.
Both arguments accept the same premise and reach compatible conclusions. The premise is that tool execution is where the wait lives. The conclusions are that the CPU should be promoted, either by coupling it more tightly or by buying more of it.
I want to take the premise more seriously than either conclusion does.
If tool execution is where the wait lives, and if the tools are what determine whether the work is actually done, then for some classes of work you do not need the accelerator in the loop at all. Not coupled to it. Not next to it. Not in it.
That is not a rhetorical move. It is a thing you can measure, and the constraint that denied me GPUs is what made me measure it.
What I ran
Thirty-nine bug fixes, reconstructed from real commits in real Python libraries. Not synthetic toys. Each task presents the codebase as it stood with the historical bug present, pinned to a repo and a commit, and the fix is judged by executable tests: feature tests that verify the bug is gone, regression tests that catch a patch that fixes one thing by breaking another, held-out reference tests the model never sees that catch a patch gaming the visible assertions, and a static gate that kills malformed output before any code runs.
The loop generates a patch, applies it, runs the checks, and executes the test suite. Every one of those steps after “generates” is tool execution. All of it is CPU and IO bound. On the rented Granite Rapids instance, all of it ran on the same cores as the model, because there was no accelerator in the loop to cross to.
The sentence I care about is the one about placement. The CPU was not orchestrating around the valuable part of that loop. The CPU was running the part that determines the outcome. The model proposes. The tests dispose. Both sat on Xeon.
The inversion
Here is where I part company with both vendor arguments.
They are arguing that the CPU deserves to be first class alongside the accelerator, because it does a large share of the timed work. I measured what happens when it is the only class, and for batch work behind a deterministic validator, the answer is that the loop closes and the work is trustworthy.
That does not refute the coupling argument. It bounds it. Coupling earns its premium on workloads where state has to move fast between the CPU and the accelerator, which is a real category: long context, multi-model serving, anything where the KV cache is the bottleneck rather than the verdict. What it does not earn a premium on is a batch of falsifiable tasks where nobody is waiting on a response and the tests decide when you are done. On that class of work, the coherent link is buying you something you do not need, and the reference price for what you do need is a CPU you have already paid for.
I could not have gotten to that framing by reasoning about it. I got there because the GPU door closed and I had to work with what opened.
The part about custom models
The commodity API floor is real and it is cheap. Sub-second first token, a couple of dollars per million output tokens. Nothing I measured on a CPU comes close to that on any per-token basis, and I am not going to pretend otherwise.
TraceLab found something about that meter worth sitting with, though. Priced at list, prefix tokens are 59.5% of what a coding-agent session costs, append tokens 29.2%, and output tokens only 11.2%. Generation is not the expensive part. Re-reading the accumulated context on every step is, even at cache-read rates a tenth of fresh input. The meter runs hardest on the part of the loop that is not thinking.
The floor also serves provider-managed models. It does not serve my fine-tuned weights. That was true when I started and it is still true. So for batch work with an executable validator, the comparison is not custom-on-CPU versus managed-on-API. The comparison is custom-on-CPU versus not-custom-at-all.
That reframes the cost advantage entirely. Nine cents per verified fix is not competing with the commodity token meter. It is pricing a workload the commodity endpoint cannot run: my weights, under my control, behind my validator. And on capacity you have already committed to, above the operating baseline you hold for availability, the marginal cost of those cycles approaches the electricity they were burning while idle.
This is a reasoning plane argument, not a compute argument. The question is not which silicon is cheaper per token. The question is where the model that encodes your specific judgment is allowed to live, and what it costs to put it to work once it lives there. The answer that fell out of a GPU denial is that it can live on the CPU you already own, do verified work on cycles that were producing nothing, and route what it cannot do to the people who were always the fallback.
Where the answer stops
I measured Granite Rapids. I do not know what any of this means on AMD or on ARM. The economic structure should travel, because committed headroom plus a deterministic gate plus a human backstop is not a statement about instruction sets. The yield will not travel, because yield is a property of the model and the substrate together, and I have already watched the same model behave completely differently across two lanes. Somebody should measure it. It will not be expensive.
I also want to be careful about a forecast I am tempted to make. On decode-dominated batch work, the sparse model was the workhorse: a twenty-six billion parameter mixture-of-experts tied a twelve billion dense model on verified yield while finishing the identical protocol in less than half the wall time. Sparse activation, not parameter count, was the fit variable. That is a measured, present-tense claim. The tempting extension is that as mixture-of-experts models get stronger, the CPU’s share of useful work grows with them. That follows, but it is not proven by what I ran, and I would rather say which is which.
And the scope limit is the validator’s scope limit. Everything above holds for work where a machine can check the answer. Bug fixes have that property. A lot of enterprise batch work has that property. Plenty does not, and for that work none of this applies.
The part I am actually looking forward to
Enterprises are good at exactly one thing that nobody gives them credit for: engineering around constraints. That is the whole discipline. You do not get the budget, you do not get the parts, the vendor’s roadmap does not match yours, and you ship anyway.
Right now every enterprise I talk to is being told the constraint is accelerator supply and the answer is to wait in line or pay for coupling. Meanwhile they are all sitting on committed CPU above their operating baseline that produced nothing last quarter, and a meaningful share of the AI work they actually need done is batch work with a checkable answer.
I did not find that because I am clever about silicon. I found it because the automation told me no in two seconds, the sales channel never called back, and I had to run on something. The constraint was the finding.