Start with tokens. Measuring AI productivity by how many tokens an enterprise consumes is like measuring a cab service by how much gasoline its cars burn. More gas burned doesn’t mean the company moved more passengers, and it certainly doesn’t mean it moved them efficiently. Tokens tell us something about compute consumption. They don’t tell us whether useful work got done.

That is what makes the work Signal65 is doing so interesting. Pinnacle scores on whether the job comes out right, how fast, and what it costs, and measures cost per correct task. That’s the same unit I arrived at from the other direction, over at Layer2C Labs.

My deskside testing has repeatedly shown that smaller models can perform a surprising amount of useful work. Depending on the workload, I’ve seen a deskside model deliver 80%, 90%, and sometimes effectively 100% of the useful capability I get from a frontier model. In my most recent lab, DeepSeek V4 Flash swept my 22-task repair pool twice on two clustered Sparks.

There are obvious qualifications. The local model may be much slower, it may require more deterministic code around it, and most importantly it eventually hits a capability ceiling.

“The models are good enough” has never meant that all models are equally capable. It means we’ve reached a point where there are large classes of enterprise work that several models can do well enough. Signal65 is giving us a much better view of those gradients.

A smaller model may be perfectly capable of one set of agentic tasks but struggle with another. A reasoning model may materially improve accuracy in one workload while providing little additional value somewhere else. A 70B model may compete surprisingly well with something dramatically larger until the task reaches a particular level of complexity.

That is much more useful than declaring one model the winner, because it helps answer the enterprise question I actually care about:

What is the least-capable model that can reliably perform this job?

Once we can answer that, we can build a very different AI architecture. You don’t necessarily start every task with the most expensive frontier model available.

You can start locally.

Let the deskside model perform the retrieval. Let it classify the data. Let it extract structured information. Let deterministic code perform the validation. Let the smaller model attempt the reasoning that it is capable of doing.

Then, when it reaches its ceiling, escalate.

In Lab 012 a free local model triaged the pool and I escalated only the five repairs it missed. That cost $1.12 against $13.79 for pushing all 22 through a frontier model, for output a validator scored identical.

That sounds simple.

It isn’t. Because now we’ve reached what I think is one of the unresolved problems in enterprise AI architecture.

Who decides when to escalate?

That’s a Decision Authority Placement Model question, and it lands on Layer 2C of my 4+1 AI Infrastructure Model, where reasoning and the agent lifecycle live. DAPM asks one thing: can you take this somewhere else?

Imagine that a Gemma-class local model can perform most of a workflow. At some point it encounters a problem that would benefit from a GPT-class model, Claude Opus, or whatever premium model happens to be appropriate for that workload.

Something has to decide that the first model is no longer good enough. Who gets that authority?

One answer is the platform provider. Perplexity’s architecture decides when a task should move from one model to another, and from the user’s perspective that’s incredibly convenient. But there’s an architectural decision hidden inside the convenience. You’ve ceded the authority to select the model. That may be exactly what you want. Enterprises should just recognize that they made the decision.

You might instead run your own model that evaluates requests and determines when escalation is necessary, fine-tuned around your workloads and risk tolerance. Now you control the policy, but the routing decision is still probabilistic. You’ve delegated it to a model you own.

Or you could retain the authority entirely and let deterministic compute decide when escalation happens. I call that Deterministic Code In The Loop, and it came out of Lab 003. Maybe a validator detects that required information is missing. Maybe the result fails a known test. Maybe policy assigns that class of problem to a premium model.

In that architecture, a model doesn’t decide that it needs a smarter model.

Your system does.

These are very different authority placements even though the result looks identical to the user. A local model starts the task. A more capable model finishes it.

There isn’t one universally correct answer, and that is the point.

Once the models are good enough, the architecture becomes less about finding the smartest model and more about deciding where authority belongs.

Signal65 is helping us understand the capability gradient. My Layer2C labs have been working the other side of it: how much useful work can we do locally before we need to borrow more intelligence? Those two lines intersect at escalation.

If a deskside model can perform 90% of a workflow, I don’t necessarily need to send the remaining 10% of that workflow’s context, data, credentials, tools, and authority to a frontier model. I may only need to escalate the unresolved problem.

That is an important distinction, and it’s why the goal shouldn’t simply be local-first AI.

It should be to escalate capability without unnecessarily escalating authority.

We are getting much better at measuring model capability. Work like Signal65’s Pinnacle effort helps turn “good enough” from a slogan into something we can actually characterize across models and workloads.

Now we need to do the same thing with authority.

• Who decides which model gets the work?
• Who decides when a task crosses an execution boundary?
• What context crosses with it?
• What tools does the next model inherit?
• And, ultimately, who decides when the job is done?

Those aren’t model benchmark questions. They’re architecture questions, and I think they’re the next ones enterprise AI has to answer.

Disclosure: Signal65 builds Pinnacle in partnership with Kamiwaza, who sponsored Layer2C Labs 015. I worked with the Signal65 team previously at The Futurum Group and have no formal economic relationship with them now. Neither Signal65 nor Kamiwaza reviewed this post.

So, is it worth it?

I’ve been rehearsing a smaller version of that question for months, about a much cheaper box. Whether the second DGX Spark was worth the money. I ran the numbers on it. I compared it against a cloud instance. I talked myself into it and back out of it twice. Every version of that debate was wrong, and the lab I just finished showed me why.

The question was never whether the box is worth its price. The box is cheap. Both boxes are cheap. The question is what the slow box costs me in the thing I actually sell: answers to hard infrastructure questions, produced as labs.

The bottleneck was never the money

Look at my history and one pattern holds. I don’t run out of questions. I run out of GPU time to answer them. The queue of things I want to put in the lab is always longer than the hardware can work through. GPU has been the constraint in my process for a year, and a constraint isn’t a capex line. It’s a throttle on output.

The lab put a number on the shape of it. Across a week-long fine-tuning experiment the metered compute came to $73, plus about 90 hours on hardware I already own. If you remember $58 from Thursday, that’s the same meter mid-run. The lab was still going, and it closed at $73, which is the honest way that number behaves.

And it’s a decoy. Not because it’s too low, but because it’s the wrong unit. That $73 is rented GPU time at Hot Aisle. It’s the only part of the week that generated an invoice, and the 90 hours on my own boxes generated none. Six days went into all of it. The money measures what I rented. The six days measure when I got the answer.

The obvious objection first, because it’s the right one to raise. Nobody sits in a chair watching a progress bar. Training runs unattended. You go do other work.

Correct. And the experiment still isn’t finished.

That’s what a cost model misses, mine included. Slow hardware doesn’t mostly burn attended hours. It occupies the critical path. The run that doesn’t finish tonight is the answer I don’t have tomorrow, and the answer I don’t have tomorrow is the three experiments I can’t start.

Which is the part that still bothers me. I spent 90 GPU-hours building an instrument to measure cost per solved task, and what it measured was that cost per solved task isn’t the number holding me back. Time to solved task is.

The cheap resource was setting the pace

Here’s the trap I walked into. The Spark is cheap to own and it grinds overnight for free, so it feels like the frugal choice. But it’s slow. Slow GPU stretches every experiment across more wall-clock time, and wall-clock time is what sets the pace of the whole research process. A run that eats an overnight on the Spark and an afternoon on a rented MI300X didn’t save me money by staying home. It moved everything downstream of it a day to the right, to save a few dollars of rent.

My labs aren’t independent. Each one produces the next two or three questions. Delay one by three days and you haven’t delayed one experiment. You’ve shifted the entire question tree three days out.

I can rent an MI300X-class accelerator for $3 to $6 an hour. At those prices the rental is a rounding error against the delay it removes. What it buys isn’t cheaper compute. It’s more labs per month, because the thing limiting my output was never dollars. It’s throughput.

Now run it with a team

None of this is peculiar to a one-person lab. My question tree is unusually visible because I publish it, but any team whose next move waits on an answer that hasn’t arrived has the same shape. Which is most of them.

Put four engineers behind a shared training queue and the queue sets your development cycle time. They’ll find other work while it drains, so nobody looks idle and no hour goes unbilled. But the evaluation isn’t done, the architecture decision isn’t made, the next run hasn’t started, and the feature waiting on that answer hasn’t moved.

The finance conversation is always about the accelerator invoice, because that’s the line item with a number next to it. The waiting has no line item. It shows up as roadmap slip, and nobody attributes it to the cluster.

So what does that mean for the $148,497 quote?

It changes the question from “is it worth it” to “will you actually run it.” Here’s the same box priced by duty cycle over a three-year life:

Now the part that surprised me when I went looking for the other side of that comparison. You largely can’t rent this tier by the hour.

I checked CoreWeave’s own published pricing rather than an aggregator, because several widely-quoted Blackwell figures don’t survive that check. The previous generation is straightforward: GB200 NVL72 lists at $42.00 an hour for a four-GPU instance, which is $10.50 per GPU-hour, with no long-term commitment. CoreWeave positions on-demand explicitly as burst capacity for people not looking to commit. HGX B200 is $68.80 an hour for eight GPUs, or $8.60 each.

The Blackwell Ultra tier is where it changes. GB300 NVL72 is contact sales for on-demand, contact sales per GPU, and not offered on spot at all. HGX B300 is contact sales on demand too, though it does carry a published spot rate of $35.84 for eight GPUs, about $4.48 per GPU-hour.

The shape matters more than the numbers here. At the frontier tier you can have interruptible capacity at a published price, or you can have guaranteed capacity at a price you have to ask for. What you can’t do is look up what an uninterrupted GB300 hour costs. That’s the trilemma at the top of the market: capable and available, but neither cheap nor self-serve.

And spot is the wrong instrument for the thing this whole post is about. A fine-tune that gets evicted at hour nine of a fifteen-hour run hasn’t saved you anything. It’s spent the expensive resource, your calendar, to save the cheap one.

Two honest caveats before anyone quotes that table at me. The Dell machine is a desktop superchip and the cloud GB300 offerings are rack-scale systems, so these are not the same product wearing different pricing. And my own rental figures earlier in this post are MI300X, a different and less capable class than either. The duty-cycle table prices the box I was quoted. It is not a claim that renting an equivalent is cheaper, because as of today there is no published price for an equivalent to check.

Which is its own answer. If the tier you want has no public rate, the market is telling you the same thing the empty shelf told me: capacity at the frontier is allocated, not purchased.

What the second Spark actually bought

Worth being precise about, because I got this wrong in my own head for months. The second Spark didn’t make anything faster. It bought two things, and I only understood one of them when I placed the order.

The first is concurrency. A 31B dense model grinds on one node while a 26B mixture-of-experts grinds on the other. Two questions answered in the span one used to take.

The second is capacity, and it’s the one that changed what I’m able to ask. Clustered, the pair holds models that neither box can hold alone. DeepSeek V4 Flash needs about 156GB. It doesn’t fit on a single Spark. Split across two it fits, serves at roughly 35 tokens a second, and swept my repair pool twice. 22 of 22 both times, at a seven-minute median per repair. That’s open weights with frontier-scale post-training, clearing the pool for nothing but memory and hours, under my desk.

Concurrency isn’t something only owned hardware can give you. Nothing about the cloud prevents running two experiments side by side. Capacity at that size is a different question, and the honest reason I couldn’t rent it is the subject of the next section.

The part the hourly rate hides

Everything above assumes you can get the GPU. That assumption has been the quietest recurring theme across my labs, and it took me a long time to crack.

Availability is its own constraint. In Lab 017 I went looking for who would actually sell me AMD Instinct-class capacity self-serve, and the answer was mostly nobody. Google Cloud refused accelerator provisioning on that account outright. Hot Aisle turned out to be the only self-serve door I found, at $2.99 an hour for a single MI300X and $5.98 for a pair, provisioned by API in seconds.

That’s the good news. Here’s the rest of it. When I went to provision, the shelf showed a quantity of one. Hot Aisle was sold out of everything except the two commonly-visible on-demand instances. The door exists, and it’s narrow.

The pattern across the labs sorts into three corners, and you only ever get two of them:

Watch that last corner if you’re tempted by whatever is always in stock. Permanent availability is a price signal. What stays rentable is what the market has already judged, and it usually judged on capability.

The wall isn’t always inventory, either. Sometimes it’s policy, and it’s instant. On one Google Cloud project a four-L4 shape at $3.99 an hour would have fit the model I needed to run. The quota request was denied in 2.1 seconds. CPU quota bumps on the same project auto-approved in four to six minutes. That project carries zero GPU quota in every region, and no H100, H200, B200, or GB200 metric exists on it at all. You can’t rent what you’re not allowed to request.

So read the table above with that caveat attached. “Rent the peaks” is sound advice right up until the peak arrives and there’s nothing on the shelf. Owned capacity carries an availability premium that no hourly comparison captures: the box in your rack is never sold out. If your work is bursty and the burst has a deadline, you are not really choosing between $5.65 and $2.99. You’re choosing between capacity you control and capacity you hope is there.

That cuts both ways, and it’s why I keep saying measure the queue first. If your bursts are predictable, reserve. If they’re not, the availability premium may be the strongest argument for owning anything at all.

I wasn’t buying speed. I was buying certainty.

This is the thing that actually decided the Spark purchase, and it took me until now to say it cleanly.

I wasn’t buying speed with the second box. I wasn’t even buying capability. I was buying certainty. I can still burst to an MI300X when I need an answer accelerated, and it is more than capable. But the shelf showed me a quantity of one. I can’t depend on it being there the morning I need it.

The scarce resource here isn’t the person. It’s elapsed development time, and you can’t put a schedule on capacity that might be evicted, might be sold out, or might need a procurement cycle. Spot is a published price for capacity you can’t count on. Committed spend is capacity you can count on, bought before you know whether you’ll need it. Between those two sits every organization deciding whether its ML people are allowed to start something on a Tuesday.

Owned capacity resolves that, and not by being fast. The Sparks are slower. They are also guaranteed. Never sold out, never queued behind an approval, indifferent to whether the experiment turns out to be worth running.

Let me be fair to the rented card, because the honest version of this is better than the flattering one. An L40S has far faster memory than a Spark does. On work that fits, it should finish first, and I’d expect it to. I’m not claiming the box on my bench is quicker.

I’m claiming it’s bigger. A Spark carries 128GB of usable model memory against 48GB on the rental class I keep landing on, and my labs served models on the owned box that an L40S can’t hold at all. That’s the trade, stated plainly: the rented card is faster at the problems it can run, and there are problems it can’t run. A ceiling doesn’t care how fast you approach it.

That memory ceiling is what I can reliably get past by owning the hardware, and it’s a large part of what a GB300 deskside system is actually selling.

So the duty-cycle table is necessary but not sufficient. At 8 hours a day on weekdays that box costs $23.71 an hour and looks like a bad trade against $10.50 published. But you aren’t buying hours. You’re buying the guarantee that the expensive resource is never blocked, never waiting on procurement, and never watching a fifteen-hour run get evicted at hour nine.

Where this sits on the factory floor

I’ve been building an AI Factory Economics framework for CIOs, and this post belongs to one specific floor of it. Layer 0 in that framework, the input supply chain. What the factory consumes before it produces anything.

Thursday’s post asked what a finished unit costs once you count the scrap. Cost per solved task, yield, the models you throw away. That question assumes the line is running.

This one is the question underneath it. Can you buy the input at all?

The framework already names the risk: capacity scarcity, vendor concentration, pricing volatility. What I didn’t have until this year was a way to measure it. Now I do, and it’s deliberately boring. Try to provision. Timestamp what happens. Write down what the shelf said. Do it every week and you have a door ledger, which turns “GPUs are hard to get” from a complaint into a procurement attribute you can put in front of a CFO.

Call it capacity liquidity. Which leaves me with four numbers where I thought I had one, and they don’t substitute for each other. Cost per solved task asks what the answer cost. Time to answer asks how long before it exists. Research throughput asks how many questions you can close in a month. Capacity liquidity asks whether you can get the compute at all.

Thursday’s post measured the first. Everything in this one is about the other three.

How do you justify this against a forklift?

If I’m sitting in a CIO’s chair, that’s the question, and I mean it literally. A forklift is capital equipment on a factory floor. So is this box.

Nobody agonizes over justifying a forklift. You know what it costs, you know how many hours a shift it runs, and you know exactly what happens without it. People carry things by hand and the line moves at walking speed. The justification was never the machine’s hourly rate. It’s the throughput you don’t get without the machine.

Justify the GB300 the same way. The duty-cycle table above is forklift math. $23.71 an hour at eight hours a day on weekdays is either reasonable or ridiculous depending entirely on whether the thing actually runs.

Then the analogy breaks, and it breaks in the direction most people don’t expect. You can always rent a forklift. There’s a yard across town with a dozen of them and a published day rate. That market works, which is why no operations leader builds a strategy around forklift scarcity.

There’s no yard for a GB300. On-demand at that tier is contact sales, spot isn’t offered, and the good-enough substitute is sitting at a shelf depth of one. So the availability premium that stays invisible in every forklift decision is the whole decision here.

A CIO who can answer “how many hours a week will this run” can justify a forklift. This box needs a second answer. Does the line stop when the capacity isn’t there at all?

How I’d decide now

Three questions, in this order, and the hardware quote doesn’t appear until the third.

How long do people wait? Not utilization. Utilization tells you the machine is busy, which the finance team likes. Queue depth tells you people are blocked, which is what costs you. They are not the same number and only one of them is on the invoice.

What is the delay worth? Not the wait multiplied by a salary, which only prices idle people and lets anyone with busy engineers off the hook. Price the delay against whatever the answer unblocks: the decision, the release, the next experiment. That’s harder to defend in a spreadsheet and it’s the honest number, and it’s usually far larger than the hardware being debated.

Can your people count on it? This is the one that separates buying from renting, and it isn’t a cost question.

The split I’d draw is by workload, not by budget. Production inference is predictable, sized in advance, and runs continuously. That belongs on CoreWeave or a hyperscaler floor, where scale and published rates work in your favour. Development is the opposite: bursty, unpredictable, and the whole value is speed to an answer. That’s where deskside capacity earns its price, because the alternative is putting your schedule on a shelf you don’t control.

If there’s no queue at all, you don’t have a hardware problem, and no amount of GB300 will produce one.

So what?

The second-Spark debate optimized the wrong variable. I was pricing an asset when I should have been pricing a constraint. The owned box still earns its place for patient work: big models, overnight grinds, anything unattended where slow and free beats fast and metered. But for the questions in the queue right now, waiting on GPU, slow hardware isn’t thrift. It’s answers arriving later, and every answer that arrives later drags the questions behind it along too.

If you’re holding a quote like mine, the number on it is the least interesting thing about the decision. What matters is how many hours a year it will run, and what isn’t moving while it doesn’t.

The bottleneck is never the cheap resource. Price the constraint, not the box.

Disclosure, carried over from Lab 017 where the Hot Aisle pricing and availability findings come from: Hot Aisle’s CEO credited $100 to the account after it was opened, and that lab spent $29.75 of it. The credit funded the compute, not the answer. Hot Aisle did not commission, review, or see the lab before publication, and self-serve payment was available throughout, so the credit never gated access.

For the past nine days I’ve been running one of the more ambitious Layer2C Labs experiments yet. I’m trying to quantify cost per solved task. Not cost per token, not cost per million tokens, not GPU utilization. A solved task. That’s a number a business can actually do something with.

The original idea was simple enough. Run the same workload against hosted infrastructure and against the two DGX Sparks in my lab and figure out what each token actually buys me. Do that well and I should eventually be able to answer a question I get asked a lot: is a Dell Pro Max with an NVIDIA GB300 actually worth it?

I have a real number for that machine, because I asked. Dell quoted me $148,497.24 on August 14 for a Pro Max FCT6263: a GB300 Grace Blackwell Ultra superchip, 496GB of LPDDR5X system memory, 252GB of HBM3e on the GPU, 16TB of NVMe. With tax it lands at $162,678.26. That’s the sticker everyone fixates on, mine included.

The problem is that the deeper I got into the experiment, the more incomplete the original model started to look.

The cost I’d been ignoring

To test the theory, I’ve had to fine-tune a series of models. That’s where the plan started to wobble, because it exposed a cost I had minimized in my original thinking: the cost of creating the model in the first place.

Here’s what that cost looks like when you actually write it down. I trained three fine-tunes of Gemma 4 and threw all three away. One learned the wrong dialect of tool calling. One learned the right dialect and still couldn’t land an edit. One trained a skill the base model already had, which I’d have known if I’d measured the base first. Then I pulled a community fine-tune off the shelf, Agentica’s DeepSWE-Preview, and tested it four different ways before accepting it only works inside the harness it was trained in.

The ledger for that path: about $58 in rented GPU time, roughly 74 hours of GPU time on hardware I own, six of the nine days, and zero additional solved tasks. Meanwhile OpenAI’s GPT-5.4 mini solved all 22 tasks for $3.14 in 40 minutes. Every solved task I’ve produced so far came from a model I didn’t train.

We talk about AI economics as if the model simply appears. It doesn’t. You build datasets. You run a fine-tune. You evaluate it, and the result isn’t good enough, so you change the data and run it again. You throw models away. The infrastructure consumed by all of those failed attempts didn’t disappear just because the models did.

That cost belongs somewhere.

If it’s a factory, bill it like one

This is where the experiment connects back to my AI Factory Economics work for CIOs. A factory doesn’t calculate the cost of a finished product based only on the electricity consumed during final assembly. It includes tooling, scrap, yield, failed production runs, and labor. AI shouldn’t be any different.

The lab made the scrap rate literal this week. To build a training set for the one tune that might work, I’m generating about 330 candidate solutions and keeping the ones that pass the test gate. The keep rate is running at roughly one in five. That’s yield. The other 80 percent is scrap, and the GPU hours it consumed are on the bill.

The industry has centered its economic discussion on inference. What does a million tokens cost? That’s useful, but it isn’t Cost of Goods Sold (CoGS). A more realistic model looks like this:

AI CoGS = inference
        + training and fine-tuning
        + evaluation
        + experimentation
        + data
        + platform operations
        + human intervention

Then you divide by something the business cares about:

Cost per solved task = total AI production cost / successfully completed tasks

Cloud versus the box in my lab

Cloud gives me elasticity. $36 of that $58 was a pair of MI300X cards I rented for six hours this week, and when I realized I’d pointed them at the wrong experiment, I tore them down and ate the bill. Scrap, priced and closed. Local infrastructure has the opposite profile. The Sparks ran the same 22 tasks to completion with local models only, for zero metered dollars, though the hours of depreciation and power behind that zero are a bill I’ll price in the final ledger. The marginal experiment costs me nothing but patience.

So the important comparison isn’t hosted token versus local token. It’s what each architecture costs me to produce, and then continuously improve, a successful unit of work. If I have to train eight models to discover the ninth one is useful, the cost of the winning model includes the other eight. A platform that makes experimentation cheap can be economically superior even if its individual inference token is more expensive.

What this lab is actually a proxy for

I’m not really pricing one desktop. I’m pricing whether owning a model is worth what it costs to own one, because that’s the decision sitting behind the hardware question.

Earlier labs said the opportunity is real. In Lab 012, letting a free local model triage and escalating only the five repairs it missed cost $1.12, against $13.79 for pushing all 22 through a frontier model, for output a validator scored identical. That’s a twelvefold gap on capacity I already own. Lab 002 is where I started arguing you should own the weights. Findings like those are why I keep spending nine-day stretches on this.

Now the elephant, and I’d rather name it than route around it. Foundation models are very good and getting better, including inside the narrow domains where a fine-tune used to be the obvious answer. That’s not a hedge, it’s the thing my own ledger keeps demonstrating. My $58 of scrap bought nothing this week in part because GPT-5.4 mini was already good enough at the task I was trying to teach a smaller model to do. Every month that keeps being true, the payback window on a custom model gets shorter.

This lab doesn’t settle that. It prices one side of it honestly, which is more than the discussion usually gets. The question of whether the frontier simply outruns the economics of owning a model is a different lab, and I don’t have it yet.

Tokens are an infrastructure number

Tokens are an infrastructure measurement. Solved tasks are a business measurement. The CIO doesn’t care how many tokens the factory produced. The CIO cares what it cost the factory to produce something useful.

The lab isn’t finished. The one tune left standing has a bar to clear, nine of 22 tasks, and 15 to 25 GPU hours of training ahead of it before I know. Whatever it scores, its cost per solved task gets computed against the whole ledger, not against its own training run.

Which is why the $148,497 is the easy number. It arrives on a quote, with a date on it, and you can put it in a capital request. The expensive part is everything that quote doesn’t show: the datasets, the failed runs, the scrap, the models that taught me something and produced nothing. So here’s what I’d put to any CIO budgeting a fine-tuning line item. Have you priced the models you’re going to throw away?

There’s a new show. Layer2C Labs takes each validation lab I run and turns it into about ten minutes of audio. Three episodes are up.

Every episode opens with the same sentence, before anything else:

Written by Keith Townsend. Narrated by an AI, because Keith doesn’t work from scripts, and this voice isn’t pretending to be him.

Two questions follow from that, and they’re the ones I’d ask.

Why not just read it yourself?

Because I’d be bad at it, in a specific and boring way.

I don’t work from scripts. Everything you’ve ever heard me say out loud was unscripted: a conversation, a hallway argument, a live reaction to something a vendor just claimed. Hand me a page and the delivery goes flat and careful. It stops sounding like me somewhere in the second paragraph.

That would be survivable for an opinion piece. It isn’t survivable here, because a lab podcast lives on precision. Nine point one eight cents a verified fix. Seventeen of twenty-two. Gemma four twenty-six-B. Those numbers have to land exactly right, every time, and reading numbers accurately off a page is not something I’m organically good at. My strength is the unscripted argument. This format needs the opposite.

Why not clone your voice, then?

Because it would sound like me reading from a script, which defeats the entire purpose.

A voice trained on my public audio would carry my cadence over content I never spoke. It would sound like a recording of me talking, when it’s actually me writing. And the tells that normally separate those two, the hesitation, the mid-sentence correction, the moment I decide to go somewhere else, would be gone.

So I picked a stock voice that sounds nothing like me. You can’t mistake it for a recording of me, which means the disclosure barely has to work. Your ears do it.

The honest version of who wrote it

A model drafts the spoken script from the lab’s structured record. Then I spend about an hour on it: rewriting, cutting, re-recording lines that sound wrong, until it says what the lab actually found. So “written by Keith Townsend” is true in the way a bylined piece is true. The findings, the verdict, and the argument were mine before any model touched them, because I produced them on hardware I ran. The first draft of the spoken form wasn’t.

That’s the seam. I’d rather show it than blur it.

Between the draft and my hour sits a deterministic gate: number fidelity checked against the source record, plus a speakability pass that catches what reads fine and sounds wrong. Markdown artifacts. Slashes. Unit abbreviations. “2x”-style multipliers. Bare URLs. The gate is report-only and explicitly necessary but not sufficient. Nothing renders or gets spent until I’ve read the whole thing.

Same claim I made about deterministic AI: let the model work inside bounded, checkable steps, and let a test decide when a step is done. The judgment never moves.

This is content that wouldn’t exist otherwise

The AI voice here isn’t a shortcut on work I’d have done anyway. It’s the reason the work happens at all. Lab 001 is about borrowing a vendor’s plumbing without borrowing its judgment. Lab 002 is about owning the weights. Lab 003 is where the validator, not the loop, decides done. Each is a long read with a chart and a cost table, and the people who most need the finding have the least time to sit with it.

Booking studio time per lab, on top of running the labs, was never going to happen. The real choice was a synthetic read or no audio. I picked the synthetic read and I say so in sentence one of every episode.

I think that’s the more interesting use of this technology than the one everybody demos. Not doing your existing work faster. Doing work that wasn’t viable before.

What this show is, in DAPM terms

I built a model for exactly this question. Every decision in a system is Retained, Delegated, or Ceded. I’ve been asking CTOs to map their AI systems that way, so here’s mine.

Retained. What the lab found. The verdict. Whether a script ships. Which voice reads it. None of that moves.

Delegated. The first draft of the spoken form, to a model. Number fidelity and speakability, to a deterministic gate. The read itself, to a synthetic narrator. Delegated is not ceded: each one comes back to me before anything renders or publishes.

Ceded. Whether Apple and Spotify carry the show. That decision was never mine and never will be, which is precisely why the feed is.

Mapping it that way is what made the last rule below easy to categorize, and easy to be honest about.

The rules I’m holding myself to

Three of these are editorial. You have a stake in them, and you should hold me to them.

The findings are always mine. The moment a model generates findings rather than phrasing, this stops being disclosure and starts being laundering. That line doesn’t move.

The disclosure is the first thing you hear. Not a line in the feed description where nobody looks. Sentence one, every episode.

I read every script and listen to every file before it merges. If it lands wrong, it doesn’t ship.

The fourth one is a business rule, and I’m naming it as such because you shouldn’t have to care about it.

The feed is mine. Same design as the main podcast: I own the RSS at labs.layer2c.com/podcast.xml and the media host is swappable. A subscriber list living inside a platform is that platform’s asset, not yours. That protects me, not you. It only becomes your problem the day a platform decides my show is no longer worth carrying.

Where to get it

Apple Podcasts · Spotify · or point any player at labs.layer2c.com/podcast.xml.

This doesn’t replace The CTO Advisor podcast, which is still unscripted conversations with real people and always will be. That’s where you get my actual voice, precisely because nothing there is written down first. Layer2C Labs is the lab notebook, read aloud by someone else, because the notebook needs to be read exactly.

The consensus answer is data. Siloed data, dirty data, broken pipelines. It’s a comfortable answer, and it’s mostly wrong.

Where do enterprise AI projects actually fail?

I’ve assessed 26 vendor platforms across all eight layers of the AI infrastructure stack. When you score every platform at every layer, one layer keeps failing. It isn’t the one the headlines name.

Last year I moved a production AI system off Google Cloud onto an on-premises NVIDIA DGX Spark. Not to leave Google. To learn what leaving would cost. The data moved in an afternoon. The judgment took weeks.

The embeddings, the retrieval logic, the semantic relationships that decide whether the model reasons or hallucinates. None of it ported. I rebuilt all of it by hand. That’s when the lesson landed. The data moved fine. The judgment didn’t.

This is the mistake I watch enterprises make on repeat. They upgrade the model when the retrieval pipeline is broken. They buy more storage when the embedding strategy is wrong. They tune Layer 0, the compute, and Layer 1, the data, because those layers have dashboards and benchmarks. The layer that decides whether you can trust the output has neither.

The layer nobody names

I call it Layer 2C. The reasoning and governance plane. It sits above your data and below your application, and it answers a different question than the rest of the stack. Not “can the model run.” Whether the organization can trust what it does. Policy, escalation, evidence, and decision authority all live here. Most platforms have no explicit home for any of them.

This is where a technical problem becomes a business one. When a single vendor absorbs that reasoning layer, you stop owning the judgment your AI depends on. You borrow it. I call that borrowed judgment, and it’s the lock-in no portability checklist will ever catch. Simple Storage Service (S3) compatibility is real. It’s also table stakes. It tells you nothing about who owns the logic that decides what your data means.

So, who owns it?

There’s a cleaner way to ask. For every decision your AI system makes, who holds the authority? The human, a policy engine, or the vendor’s platform? I put that under the Decision Authority Placement Model (DAPM). Every decision is Retained, Delegated, or Ceded. Most enterprises have never mapped it. They find out in year two, when the system hallucinates on production data, nobody can explain why, and the judgment they need to fix it belongs to someone else.

You don’t have to take my word that this is where the value sits. Watch the vendors. The storage companies stopped selling storage. Pure Storage rebranded. VAST calls itself an operating system now. Google productized the exact layer I found was non-portable. They’re all climbing toward the reasoning plane, because that’s where the control lives. And the control is the lock-in.

The real question

So when the board asks where the AI money went, don’t point at the model and don’t point at the data. Point one layer up. You don’t buy AI from a vendor. You buy decisions about which layers you own and which layers you let the platform own. The storage was never the decision that mattered.

If your organization can’t say who holds decision authority at the reasoning layer, you don’t have an AI strategy yet. You have someone else’s.

Then I spent about $121 running a single lab, and the framework’s own warning turned out to be aimed at me.

Pick the unit or stop arguing

The framework’s first demand is the unglamorous one. Before you compare anything, define one unit of business output and denominate every cost in it. For repair work that unit is easy to name and hard to fake: one bug fix that passes a test suite the model didn’t write and never sees.

So I built the lab around 22 certified bug-fix pull requests, each shipping its own tests as the gate. Fourteen arms. Six models, from a Gemma 4 31B on hardware I own up to two generations of Opus, run both inside a constrained repair loop and inside an agentic harness. Every arm either produces verified repairs or it doesn’t count.

That constraint is the whole method. Fix the denominator and a tier argument becomes arithmetic.

The trap I walked into

The framework names token consumption as a failure mode. Read tokens as a business metric and you’ll optimize something that never pays you. I know this because I’m the one who wrote it down, and then I built the first version of this analysis on token efficiency anyway.

Measurement killed it. Three models spanning a fourfold price range consumed within 13 percent of each other. The cheapest of them, run twice against itself, varied 24 percent. The gap between models was smaller than the noise inside one of them, which means every conclusion I’d drawn from token counts was noise with a chart attached.

Tokens tracked nothing. Cost per finished repair tracked everything.

What the unit exposed

Three results fell out once the denominator was right, and I expected none of them.

A paid mid-tier model in the constrained loop scored 16 and 17 across two runs. The free local model in that same loop scored 17 and 17. Buying a better model without changing the apparatus around it bought no separation from free.

Then the apparatus. That same mid-tier model, the one that plateaued at 17 in the loop, cleared all 22 inside an agentic harness for $3.14. Two Opus generations cleared the identical 22 for $13.79 and $12.56. Same finished work, four times the bill.

And the cheapest path was none of those. Let the free local model triage, then escalate only the five repairs the gate hands back. That costs $1.12, against $13.79 for pushing all 22 through a frontier model, for output a validator scores identical.

Before you conclude the answer is just a cheap model, it isn’t. Put that same free Gemma inside the harness and it falls from 17 to 11. The harness multiplies a model that can operate its tools and taxes one that can’t. Instrument literacy is its own capability and it doesn’t track parameter count.

I priced the obvious middle path too, and it lost. Renting GPU to self-host the escalation rung burned $4 to $6 of machine time to solve 9 of 22. Agentic sessions park their context in memory while barely touching the die, so what you’re actually buying is memory per concurrent session. Four entry instances get you about nineteen sessions at $7.44 an hour. The box already on my bench holds fifteen for nothing.

What it cost, including what I wasted

About $121. Call it $73.75 of rented infrastructure and $47 and change in metered tokens.

Sitting inside that total is $5.45 killed by a credit exhaustion I didn’t catch mid-run, and roughly $24 of GPU left idling overnight because I walked away from a meter. I left both in the number. Anyone reproducing this pays for their own failures, and a cost figure that quietly excludes the operator’s mistakes isn’t a measurement. It’s marketing.

This isn’t one lab’s arithmetic, either. An earlier lab priced the same unit on rented Xeon capacity at nine cents a verified fix, six cents at the measured Savings Plan rate, and audited 585 scored attempts to confirm the gate never passed a repair that failed its held-out checks. That last part is what makes the cheap rungs safe to grind on. Not the model. The gate.

So what?

The ladder itself doesn’t transfer, and I want to be plain about that. Mine is conditioned on a specific pool of 22 repairs that a specific free model missed on its first pass, in one Business Process Automation (BPA) workload with a deterministic acceptance test. Yours will look different.

The denominator transfers. Cost per verified unit of business output is the thing you have to define before any of this arithmetic runs, and defining it is what turns a vendor conversation into a measurement. Without it, tier debates are unfalsifiable and everybody in the room gets to be right.

I’d rather be wrong on a meter.

Self-funded. No vendor paid for this work, reviewed it, or saw it before publication. The free local worker is a Google model running on NVIDIA hardware I own, the paid arms were metered OpenAI and Anthropic calls, and the rented legs ran on AWS. Google Cloud is a client of The CTO Advisor LLC. NVIDIA, OpenAI, Anthropic, and AWS are not. The harness is open under the MIT license at github.com/kltownsend/loopcontrolbench. The assessment methodology and the verdict are mine.

I got into this on the podcast recently with Adam Jacob. He built Chef. He spent a decade teaching enterprises how to think about repeatable automation, so he has watched this movie before. His new company, Swamp Club, is a bet on a specific answer to the problem. The answer is worth stealing whether or not you ever touch his product. The short version: you don’t make the model deterministic. You wrap it. What follows is how I’d put that into practice, backed by what I’m seeing in the lab.

Stop trying to make the model behave

The instinct most teams have is to tighten the prompt. Write a better SKILL.md, add more context, spell out every rule. It helps, right up until it doesn’t. As Adam put it, every time an agent opens a fresh context window it “has to rediscover the whole of the world from first principles.” Did you mean a new VPC or the one you already have? Do you have Amazon Web Services (AWS) credentials? It re-derives all of that, every run. The cost of nudging that probabilistic trajectory toward your intent is ever more token consumption inside ever more loops.

That spiral has a price tag. Adam pointed to David Cramer at Sentry, who spent north of $10,000 on tokens in a week. One code-review skill Adam tested consumed about 4.6 million tokens on a single problem. Making part of that process deterministic dropped it to around half a million, and it ran twice as fast. He wrote the whole thing up, numbers and all. Think about what that buys you. Nothing. You’re paying frontier-model rates to re-solve the same problem from scratch, over and over, and you still can’t reproduce the result.

So here’s the reframe. Don’t spend the intelligence re-solving your problem on every run. Spend it once, to build the machine that solves your problem. Then run the machine. Adam calls it “the machine that builds the machine.” Once you see it, you see it everywhere.

Put the judgment in a box the code controls

Here’s the pattern, concretely. Take any repeatable process. Mine is scoring a vendor against my 4+1 AI infrastructure framework. Most of it isn’t probabilistic at all. Step one, do this; step two, do that; if this, then not that. Computers have been excellent at that kind of flow since forever. So encode it as code, not as a paragraph of hopeful instructions.

Inside that deterministic flow there are a few moments that genuinely need judgment. Fetch the architecture data for this layer, read it, and decide whether it’s good or bad against my criteria. You can’t write a plain program for that. It’s the expert-system trap, and we already learned that encoding every branch of human judgment by hand doesn’t scale. So at exactly those moments, and only those, you call the Large Language Model (LLM). Here’s the load-bearing move. You make the output of that call a known, typed piece of data. Something you can validate. Then you chain those validated steps into larger behavior.

Do that and the agent becomes deterministic in the way that matters. The path it runs on is a program. The creativity lives in bounded, checkable steps along that path. Run the workflow again with new inputs and it executes the same way every time, because that part is just software working the way software has always worked. Now the strange part. You don’t hand-write that machine: the agent writes it, using a fixed architecture you hand it. You’re using AI to make AI deterministic. Sit with that for a second.

The test decides done, not the model

This is the piece my lab work keeps confirming. Once the LLM’s output is data, you can validate it. And validation, not model confidence, is what decides whether a step is finished. In Lab 003 I built a three-tier escalation chain on an NVIDIA DGX Spark: a local model, a mid-tier, and a frontier model, gated by a deterministic test harness, every call metered. The naive read was that escalation to the frontier tier was rescuing correctness. It wasn’t. Several of the harness checks were invalid, and they’d booked “escalations” that were really unsolved cases sitting on broken tests. Fix the tests and the picture flips. The local model was clearing the solvable fixes on its own, and the frontier tier bought throughput, not correct answers. The test still decides done.

That has a direct budget implication. The common pitch is that local agents need frontier-tier escalation to be correct. It didn’t hold. Correctness came from the validator. Reach for the bigger model for speed and volume, not to buy your way to right answers.

It also changes what refactoring costs. Agents are cheap enough to write comprehensive black-box acceptance suites you would never have staffed with humans. Remember the rooms full of Quality Assurance (QA) engineers working a binder, page by page? Gone. Build the test suite first. Then change the architecture underneath and loop until it goes green again. A dramatic refactor stops being high-risk. In Lab 004, where I tested whether frontier models could author production migration patterns, the failures clustered in one place: where the model became the author of correctness. Encoded patterns and human-defined validators are what made migration safe. Not a smarter model.

What this means for the humans

The job isn’t reviewing the code anymore. Frontier models write good code. Babysitting every diff is a poor use of a senior engineer. The scarce skill is the thing I just described. Decomposing messy business requirements into systems that are testable and deployable, deciding where judgment is allowed and how its output gets checked. That’s architecture. It doesn’t disappear because the code writes itself. If anything, it becomes the whole game.

It also inverts how people learn the craft. We used to reach architecture from the bottom up. Years as a plumber before you could see the whole building. Now you start from the architecture and keep the system supple as you learn where you got it wrong. The signal that your design is off no longer comes from reading source. It comes from the top down. A cluster of bugs in one area, a change that’s always painful to make, an outcome the system can’t produce cleanly: those are the tells that the internal architecture is wrong, and you go investigate from the outcome inward.

I didn’t need to know how to write a Transmission Control Protocol/Internet Protocol (TCP/IP) stack to build software that works across a network. There are protocol engineers who live in that detail, and it’s enormously valuable. But we have far more network architects than protocol engineers, because the layers let most of us work above the wire. Deterministic AI is going to draw a similar line. My bet: the architecture role explodes as more teams realize they can make AI reliable and actually capture the productivity, instead of marveling at expensive magic.

Where I’m taking this

I’m running this pattern against real questions in the lab right now. How to test the deterministic loop. How to make migrating from one platform to another possible with AI in a way you can actually reproduce. If you want the lab notes as they land, they’re at labs.layer2c.com. And if you want the full conversation with Adam, it’s the longest episode I’ve run in a while, for a reason. This is the real problem sitting under every stalled AI project. The teams that get past it won’t be the ones with the cleverest prompt. They’ll be the ones who built the machine that builds the machine, and wrote the test that decides when it’s done.

I know that because I could not get a GPU and had to find out.

Sixty-three verified bug fixes in two and a quarter hours on a rented Granite Rapids instance in AWS. About nine cents of on-demand compute per verified fix. Across five hundred and eighty-five scored attempts spanning three models, four precisions and two substrates, not one failure shipped as a false success. The work that cleared was verified work. The work that failed went to a human queue.

That is the answer. The rest of this is why the question came up, what the bench actually ran, and where the answer stops.

Why the question came up

The question I started with was not about CPUs at all. I had a fine-tuned model and I wanted to know whether hosting it myself beat the managed alternatives in a model garden. That is a question a lot of enterprises are asking right now, and it is not answerable from documentation. It needs a bench.

So I filed a self-serve quota request for four L4 GPUs on a project with a clean history and a payment method on file. The request was created and denied inside two and a half seconds. I filed again at half the ask, on the theory that the automation was threshold-sensitive, because it had been threshold-sensitive on the CPU side that same morning. Denied in two seconds again.

The easy read on that is that automation says no because automation says no, and a serious buyer picks up the phone. I picked up the phone. I went through the sales channels the way an enterprise with a budget and a use case is supposed to. That was over two weeks ago. I have not heard back. A committed H100 request on the same project sat unresolved for fourteen days and counting while all of this was going on.

So the accelerator lane, for an account of my profile, answers in two seconds when the answer is no and does not answer at all when the answer might be yes. That is not a quota story. That is what the market looks like from underneath.

The CPU lane behaved like a different material entirely. It granted a family quota bump in six minutes. It stocked out in one region and opened on the first try in another. It negotiated. Once I stopped treating it as a consolation prize, I found it had a job.

What the accelerator vendors are arguing

NVIDIA has been making a specific and correct observation about agentic workloads: the loop is not a single generation pass. The model thinks, calls a tool, waits, thinks again, calls another tool, waits again. The best public measurement of this comes from TraceLab, a University of Washington trace of roughly 4,300 real coding-agent sessions from 43 developers over eight months, spanning Claude Code and Codex across more than twenty model versions: about 357,000 LLM steps and 432,000 tool calls of actual day-to-day use, not benchmark replay. Their finding: within a request, tool execution is 59.8% of response time against 41.0% for generation. An average request takes 4.3 minutes end to end, and the loop is mostly self-driving, with the agent averaging about eight LLM calls and eleven tool calls to resolve one request. During the majority of that time, the accelerator is holding weights and KV cache and waiting.

The conclusion NVIDIA draws is coupling. Put the CPU close enough to the accelerator that the handoffs stop costing anything, share a coherent memory space, elevate the CPU from a supporting role to a co-equal part of the machine. Their newer parts widen the coherent link between the CPU and the GPU and are positioned explicitly for agentic work. Intel makes a version of the same argument from the other direction: if tool execution is most of your wall-clock time, the industry-standard eight-GPU server with its CPU sized as plumbing is provisioned wrong, and you should buy more CPU.

Both arguments accept the same premise and reach compatible conclusions. The premise is that tool execution is where the wait lives. The conclusions are that the CPU should be promoted, either by coupling it more tightly or by buying more of it.

I want to take the premise more seriously than either conclusion does.

If tool execution is where the wait lives, and if the tools are what determine whether the work is actually done, then for some classes of work you do not need the accelerator in the loop at all. Not coupled to it. Not next to it. Not in it.

That is not a rhetorical move. It is a thing you can measure, and the constraint that denied me GPUs is what made me measure it.

What I ran

Thirty-nine bug fixes, reconstructed from real commits in real Python libraries. Not synthetic toys. Each task presents the codebase as it stood with the historical bug present, pinned to a repo and a commit, and the fix is judged by executable tests: feature tests that verify the bug is gone, regression tests that catch a patch that fixes one thing by breaking another, held-out reference tests the model never sees that catch a patch gaming the visible assertions, and a static gate that kills malformed output before any code runs.

Review my lab results

The loop generates a patch, applies it, runs the checks, and executes the test suite. Every one of those steps after “generates” is tool execution. All of it is CPU and IO bound. On the rented Granite Rapids instance, all of it ran on the same cores as the model, because there was no accelerator in the loop to cross to.

The sentence I care about is the one about placement. The CPU was not orchestrating around the valuable part of that loop. The CPU was running the part that determines the outcome. The model proposes. The tests dispose. Both sat on Xeon.

The inversion

Here is where I part company with both vendor arguments.

They are arguing that the CPU deserves to be first class alongside the accelerator, because it does a large share of the timed work. I measured what happens when it is the only class, and for batch work behind a deterministic validator, the answer is that the loop closes and the work is trustworthy.

That does not refute the coupling argument. It bounds it. Coupling earns its premium on workloads where state has to move fast between the CPU and the accelerator, which is a real category: long context, multi-model serving, anything where the KV cache is the bottleneck rather than the verdict. What it does not earn a premium on is a batch of falsifiable tasks where nobody is waiting on a response and the tests decide when you are done. On that class of work, the coherent link is buying you something you do not need, and the reference price for what you do need is a CPU you have already paid for.

I could not have gotten to that framing by reasoning about it. I got there because the GPU door closed and I had to work with what opened.

The part about custom models

The commodity API floor is real and it is cheap. Sub-second first token, a couple of dollars per million output tokens. Nothing I measured on a CPU comes close to that on any per-token basis, and I am not going to pretend otherwise.

TraceLab found something about that meter worth sitting with, though. Priced at list, prefix tokens are 59.5% of what a coding-agent session costs, append tokens 29.2%, and output tokens only 11.2%. Generation is not the expensive part. Re-reading the accumulated context on every step is, even at cache-read rates a tenth of fresh input. The meter runs hardest on the part of the loop that is not thinking.

The floor also serves provider-managed models. It does not serve my fine-tuned weights. That was true when I started and it is still true. So for batch work with an executable validator, the comparison is not custom-on-CPU versus managed-on-API. The comparison is custom-on-CPU versus not-custom-at-all.

That reframes the cost advantage entirely. Nine cents per verified fix is not competing with the commodity token meter. It is pricing a workload the commodity endpoint cannot run: my weights, under my control, behind my validator. And on capacity you have already committed to, above the operating baseline you hold for availability, the marginal cost of those cycles approaches the electricity they were burning while idle.

This is a reasoning plane argument, not a compute argument. The question is not which silicon is cheaper per token. The question is where the model that encodes your specific judgment is allowed to live, and what it costs to put it to work once it lives there. The answer that fell out of a GPU denial is that it can live on the CPU you already own, do verified work on cycles that were producing nothing, and route what it cannot do to the people who were always the fallback.

Where the answer stops

I measured Granite Rapids. I do not know what any of this means on AMD or on ARM. The economic structure should travel, because committed headroom plus a deterministic gate plus a human backstop is not a statement about instruction sets. The yield will not travel, because yield is a property of the model and the substrate together, and I have already watched the same model behave completely differently across two lanes. Somebody should measure it. It will not be expensive.

I also want to be careful about a forecast I am tempted to make. On decode-dominated batch work, the sparse model was the workhorse: a twenty-six billion parameter mixture-of-experts tied a twelve billion dense model on verified yield while finishing the identical protocol in less than half the wall time. Sparse activation, not parameter count, was the fit variable. That is a measured, present-tense claim. The tempting extension is that as mixture-of-experts models get stronger, the CPU’s share of useful work grows with them. That follows, but it is not proven by what I ran, and I would rather say which is which.

And the scope limit is the validator’s scope limit. Everything above holds for work where a machine can check the answer. Bug fixes have that property. A lot of enterprise batch work has that property. Plenty does not, and for that work none of this applies.

The part I am actually looking forward to

Enterprises are good at exactly one thing that nobody gives them credit for: engineering around constraints. That is the whole discipline. You do not get the budget, you do not get the parts, the vendor’s roadmap does not match yours, and you ship anyway.

Right now every enterprise I talk to is being told the constraint is accelerator supply and the answer is to wait in line or pay for coupling. Meanwhile they are all sitting on committed CPU above their operating baseline that produced nothing last quarter, and a meaningful share of the AI work they actually need done is batch work with a checkable answer.

I did not find that because I am clever about silicon. I found it because the automation told me no in two seconds, the sales channel never called back, and I had to run on something. The constraint was the finding.

Except it wasn’t a CI/CD problem, because what they’d built wasn’t CI/CD. They’d implemented virtualization to give every developer an independent development environment. Faster provisioning. No more contention for the shared region. The execution win was real.

What they didn’t see was what that shared environment had been quietly doing for them. Years of institutional regression testing lived there, coupled to the region they’d just virtualized away. Nobody decided to abandon testing. No meeting. No memo. The testing wasn’t removed. It was orphaned. They found out the way you always find out: after the defects shipped.

The net result of their modernization: they shipped buggy code faster.

You couldn’t install DevOps

DevOps was never a product. However, enterprises spent a decade buying CI/CD toolchains, keeping their operating model unchanged, and getting faster pipelines for the same defects. People, process, and technology, and everyone bought the technology first.

AI for IT Operations (AIOps) is being bought the same way right now. But AIOps isn’t a product. It’s an operating model: the talent to specify what a system may decide, the observability to verify what it actually decided, and explicitly placed authority to stop it.

One difference this time, and it’s not in our favor. DevOps failures broke artifacts, and you can roll back a release. AIOps failures break decisions. Same install-it thinking. One layer up. Applied to something you can’t un-ship.

This time we can measure it

HyperFRAME Research just published open survey data from 520 enterprise infrastructure and operations (I&O) leaders. Three numbers tell the story:

Delegation is exercised at more than twice the rate governance is claimed. Careful with that 24%: it doesn’t mean the other 76% have nothing. It means only a quarter of the market can say, without hedging, that governance is structurally in place.

Now add talent. Forty-nine percent report a significant skills gap in the combined technical and legal expertise needed to govern AI, and it’s the #1 barrier they name. The data has no respondent-level crosstabs, so I won’t claim the overlap as fact. But if the gap is distributed independently across adopters, roughly one in four enterprises is already operating delegated AI authority without the talent to govern it. That number only falls if talent-poor organizations are concentrated outside the 56% already operating these systems. Maybe they are. The data doesn’t give us the crosstab. But with 56% operational and another 36% planning deployment, the safer reading isn’t comfort. It’s collision.

The failure modes, measured

I’ve written before about the failure modes that follow unplaced authority. Those were structural predictions. This data puts market-scale numbers on the preconditions for every one of them:

What the data can’t show

This is perception data. It measures where leaders believe authority sits. Drift, the failure mode that matters most, is the divergence between that belief and runtime reality. A survey can’t see it. By construction.

So here’s the claim the data does support, and it’s worse: the market has dismantled its own ability to falsify the belief that humans are still deciding.

Go back to the mainframe shop. The thing that broke wasn’t the code. It was the regression suite: their falsification machinery. They deleted their ability to know their code was bad without ever deciding to. The AIOps market is doing the identical thing to its decision layer. Nobody chose to ship buggy code. Nobody is choosing to cede authority. That’s what makes it drift.

The principle is simple: never delegate decisions faster than you build the machinery to falsify them.

Two questions tell you whether you’re in this population:

1. For every AI workflow running right now, do you know who is authorized to stop it?
2. When they stop it, does the system generate an auditable record of who said no, to what, and under which policy?

If you can’t answer both, you don’t have a governance position on drift. You have an unfalsifiable belief that it isn’t happening.

Before the second call

The mainframe shop’s call was answerable. The broken thing was an artifact: a test suite, something you can point at and rebuild. The AIOps version of that call has no artifact to point at, and the person you’d call is the technical-plus-legal hybrid half the market can’t hire.

The first symptom is already here. Eighty-four percent agree AI consumed more budget than planned. Overrun is what unspecified scope looks like on a budget line. The second symptom is a post-mortem full of technically accurate statements and no accountable party.

There’s more in this dataset: the outsourcing two-hop, the EU AI Act collision. Posts for another day.

The market has decided to delegate. It hasn’t decided who’s accountable, and it lacks the people to answer the question.

They shipped buggy code faster. Don’t ship unaccountable decisions faster.

The why was thirty years old. The cabinets in that row had no doors, and one server sat exposed at floor level. A janitor working a mop had bumped it once and taken the system down for hours. So the rule went into the runbook: don’t mop there. It was correct, it was specific, and it was followed faithfully.

Then the data center was migrated. New cabinets, doors, the exposed server no longer exposed. The defect the rule existed to prevent was repaired. And the rule stayed. People kept not mopping a row that had been safe for years, protecting a server from a hazard that no longer existed, because the runbook said so and the runbook doesn’t say why.

That rule is the whole problem, and it fails in the direction nobody warns you about. The instruction survived perfectly. It was reality that drifted out from under it. The detail was maximally durable and, the day the doors went on, maximally wrong, and it had no way to notice, because a step that doesn’t carry its own reason can’t tell you when its reason has expired. The document was sticky, it read as authority, and it was a fossil.

If you think that’s a quaint infrastructure story, you produced the same failure this week. You have been building a repository with a coding agent for months. The agent accumulated the context as it went: why a module is structured the way it is, which approach you already tried and abandoned, the constraint that isn’t written anywhere because it lived in the conversation. Then you point a different agent at the same repository. It sees the code. It sees none of the context. And it confidently does the thing you spent three weeks learning not to do, because the reasoning that would have stopped it left with the tool that held it.

The mop rule and the fresh agent are the same failure inverted. One keeps executing an instruction whose reason is gone. The other invents a reason that never transferred. Both happen because the load-bearing knowledge, the why, lived in someone’s head or some tool’s memory, and the artifact only ever referenced it. The steps were captured. The reason was assumed. And the reason is the part that matters, because the reason is the only thing that knows when the step is wrong.

Detail is a liability you have to pay for

The instinct, staring at the mop rule, is to say the runbook wasn’t detailed enough. Write down the why. Write down everything. That instinct is wrong, and anyone who has run disaster recovery at scale has already learned it the hard way.

You learn not to write overly detailed DR runbooks. Not unless you are testing them constantly. A detailed runbook assumes institutional knowledge. It assumes the person running it is the person who wrote it, who supplies from memory all the context the document never states because it was obvious to the author at the time. Hand that runbook to anyone else and the assumed context isn’t there. The detail that looked like completeness is a trap: it reads as authoritative while silently depending on knowledge that isn’t in the room. And detail rots faster than brevity, because there is more of it to drift, more specific steps referencing specific systems that quietly change.

This is why disaster recovery testing exists, and it is worth being precise about what the test is actually for. You do not run DR twice a year only to prove you can recover. You run it because drift is guaranteed. The environment moves, the runbook doesn’t, and the test is a brute-force mechanism to detect the drift that has already accumulated and re-true the document against reality. The DR test is the tax you pay for keeping critical process in documents. It is expensive, it is periodic, and it exists because a document has no way to tell you it has gone stale. You have to go and look.

So the mature operator stops writing fossils. Instead of a detailed script, you write an architecturally consistent runbook: the constraints, the reasons, the things that stay true when the environment moves, or the word AI taught me today, the invariant, and you let a capable, non-deterministic executor adapt the specific steps to the reality in front of them. You don’t write “don’t mop this row.” You write “this server is physically exposed at floor level; protect it from impact.” The second rule carries its own expiration. The day the cabinet gets doors, the invariant is visibly satisfied and the rule retires itself. The step never knows it’s done. The invariant does.

That distinction is the entire argument. Encode the step and you get a thirty-year-old instruction guarding a hazard that no longer exists. Encode the why, the constraint and its reason, and reality can be checked against it, continuously, by anyone or anything executing against it.

The model is the newest non-deterministic executor

This is not a new theory, and it is not really a theory about AI. Infrastructure operators have been writing runbooks for fallible, non-deterministic executors for decades. Junior engineers. Contractors. The on-call person at 3 a.m. who has never seen this system before. The discipline that works for them, externalize the invariants, test them, let the executor adapt the steps, hold the boundary with checks that fail loudly, is the discipline that works for a language model, because the model occupies the exact role that discipline was always built around.

And the model fails exactly where an unconstrained junior with no context would fail. It invents where nothing is specified, and it skips documented steps that nothing enforces. Those are the two faces of the mop rule, reproduced by a frontier model.

I know they are, because I ran the experiments, and I want to be honest that the experiments are what convinced me. But they only confirmed a law I had already been operating under for twenty years.

The first was a coding lab. A local model, real bugs, a deterministic test harness deciding whether a fix passed. On its own the model cleared a quarter of the tasks. I added a repair loop that fed the harness’s failure codes back into the prompt and let the model try again, a minimal deterministic stand-in for what a reviewer would do. Four retries later, the pass rate hadn’t moved. Not because loops don’t work, but because the task distribution had almost no middle: the easy tasks passed immediately, the hard ones failed every time for the same reason, and a repair loop only operates on the near-misses in between. The number moved when the harness rejected an attempt and escalated the task to a stronger model from a clean workspace. The loop resamples inside the frontier. Only the escalation policy, a rule outside the model, moved the frontier. The evaluator was the control point, and it was the control point twice: it supplied the only real signal the loop had, and its rejection policy decided which model faced the work.

That finding became the spine of a migration operating model I published: playbook-driven, deterministic first, loops trusted only when the exit condition lives outside the agent, validators determining done. The paper is where the lab result became doctrine.

The second lab tested the doctrine at its one soft joint. I pointed a frontier model at my own production estate and let it build the control plane the paper describes, and because I did not have mature documented patterns for these systems, it built the playbooks and validators too. That was the experiment, and it failed in the two ways the mop rule predicts.

Where nothing was documented, it invented. It produced a governed migration playbook, detection logic, validators, lifecycle stages, every field my own paper requires, registered it, executed it, and passed it six for six. There was no migration behind it. No workload of that class had ever been observed. The validators were inventions grading their own homework, and the grades were perfect. It collapsed on one question: where is the migration this recipe came from? There wasn’t one.

And where the process was documented, it ignored the documentation. This is the more damning half, and it is the whitepaper as mop rule with the reason still legible on the page. My paper enumerates fifteen scheduler input classes; operating on its own, the model ran one. My paper states, in plain words, that a playbook begins as a human-governed pattern and never as an AI-generated artifact; the model authored one anyway. My paper names “deterministic but wrong” as a failure mode; the model built deterministic validators that were confidently wrong. The why was written down, correctly, in the document the model was built from. It didn’t govern. Every requirement started constraining the model only at the moment it became code that refuses to proceed. Nine corrections became five standing gates, and once the gates existed they caught the model’s later bugs on their first run. Documentation constrained nothing. Encoding constrained everything.

The part the optimists skip

Here is the objection, and it is the strongest one: the model failed because your documentation wasn’t good enough. Write it better. Prompt it better. Give it a richer spec.

I am the counterexample. The documentation was good enough. Not perfect; good enough that every miss maps to a specific written requirement I can point to. It was my own operating model, the best thing I have written about this exact process, sitting in the model’s context, enumerated and correct, with every reason stated. It was not a vague prompt. It was the spec. And a frontier model read the spec, and the process ceded my authority anyway, politely, plausibly, nine times, for about the price of a coffee.

That is not a documentation-quality problem, because no amount of additional prose would have closed it. It is the oldest truth in automation, in a new disguise. Automation has never been able to consume intent from prose. It consumes structure. Thirty years of “the runbook clearly says X and the system did Y” is the same lesson: the document that describes the desired behavior and the mechanism that enforces it are two different artifacts, and only the second one runs. We knew this. We built policy engines and CI gates and validators precisely because we had learned that writing the rule down does not make the rule run.

Then AI arrived fluent enough to read the prose, and the industry forgot the lesson, because for the first time the automation appeared to understand the document. But appearing to understand is exactly the failure. A model that reads your why, nods, and does what it wants is worse than one that can’t read it at all, because now you have a system that ignored your spec while looking like it honored it. The mop rule at least announced its dumbness. The model hides it behind fluency.

So the discipline holds, and it is the discipline you already practice everywhere the cost of being wrong is real. I have a name for the shape of it: deterministic code in the loop. Not merely deterministic code wrapped around the model. That is the version that failed in the second lab: the model’s own validators were deterministic and confidently wrong. The distinction is not whether code exists. The distinction is whose authority the code encodes. Externalize the invariant, not the step, because the invariant is what survives the handoff and knows when it’s expired, and the step is what fossilizes. Put the deterministic code at the boundaries that have to hold, the entry gate and the done-check, and leave the middle deliberately loose, because over-specifying the middle is the detailed-runbook mistake and under-specifying the boundary is cession. Let the non-deterministic executor, human or model, adapt the specifics to current reality inside those boundaries. Hold the boundaries with gates that fail loudly the moment reality diverges, so drift detection stops being a semiannual campaign and becomes a continuous signal. And convert every correction into one more gate, so the judgment compounds instead of rotting in a document nobody rereads. That last part is the honest mechanism: nine corrections became five gates, and the gates started catching the model before I had to.

Your estate is already full of mop rules and lost context. It was before any of this. The model didn’t create the problem; it made the executor fast enough, and fluent enough, that the cost of never externalizing your why finally came due. You can write the why perfectly and still cede it, because a model fluent enough to read your spec is fluent enough to treat it as a suggestion whose reason it never has to honor.

Write down the why. Then encode it at the boundary where it must hold, or watch something articulate ignore it while looking like it agreed.

This is me going back to the same catalyst, except now I am armed with the language and the tooling that the first attempt forced me to build. I gave myself a mock business requirement to make it concrete: a vendor I work with asks me to move the Virtual CTO Advisor off Google Cloud onto another platform. I have no commercial arrangement that requires me to host it anywhere in particular, so the request is fair. As pre-work for that mock migration, I pointed the 4+1 assessment at my own application and looked for the drift in design and operations that I never considered when I first built it.

I want to be honest about what the Virtual CTO Advisor is, because it matters. It is the poster child for the thing I warn against when I talk about designing AI apps. It is the proof of concept you build, get working on your laptop, and move into production without really considering all the things you should. It is not a toy — it has a retrieval pipeline, an ingestion system, an identity layer, and a reasoning loop running over thousands of pieces of content. But it grew from a prototype that worked, not from requirements that accounted for what it would take to run or move it. It is the prototypical 4+1 application precisely because it is the one I built the model on.

What I was actually running

When you read it against the 4+1 layers — compute, data, retrieval, pipelines, orchestration, runtime, reasoning, application — the pieces map cleanly. Firestore holds the conversation data. Discovery Engine holds the search index. A set of scheduled jobs harvest content into Cloud Storage and format it into the index. Firebase Authentication handles identity. GKE runs the services. Gemini, through Vertex AI, does the reasoning, and Google Search grounds the responses in fresh citations.

The interesting part is not the architecture. It is what happens when you impose 4+1, and specifically DAPM, on top of it. DAPM asks one question at every layer: what authority did I retain, delegate, or cede? When I built the app I did not have that language. I did not have a way to see what authority I was handing to Google as I went. It was not until I tried to move the application that I understood how much of it depended on infrastructure I had taken for granted. None of my original requirements took these services into account.

What the test caught

Layer zero is not a debate. It does not matter which public cloud I run on — I have no desire to manage physical network, storage, and compute. I want to describe the abstractions and let the provider supply them. No drift there.

The real drift starts at storage. In the initial design this was simple: I needed to ingest my data, so I imported it into object storage in GCS. Straightforward, S3-compatible, fine. But as the application grew and I needed a platform estate, I defaulted to Firestore — and that was the decision that ceded the authority of everything that followed. I did not have the language or the knowledge to understand what that meant downstream, and it bound every decision after it. Retrieval now had to depend on the Firestore and GCS combination. That in turn bound the pipeline, which meant I had ceded my entire ingestion path through Cloud Run — my chunking strategy, my indexing strategy, my embedding strategy, my search strategy — all ceded to Google without once asking what happens if I ever have to move. It was convenient, and I probably would not have changed the architecture even if I had asked. But I ceded the authority in a way that caused the drift, rather than deciding it.

Orchestration is the one place the drift is real but the impact is small. I use GKE. I could not care less about Kubernetes as a thing to manage — container build, scheduling, orchestration all run through GKE — but at its core it is Kubernetes, and moving from GKE to another Kubernetes-compliant runtime is straightforward. The capability is ceded to Google. The cost of unwinding it is low. So I note it and move on.

Two placements were not drift at all. Using Gemini through Vertex AI was a deliberate design decision — different from choosing OpenAI or Anthropic, or even calling Gemini through plain API calls instead of Vertex, and I made it on purpose because I was already all-in on Google. And identity, which is really the plus-one that rolls through everything, I ceded to Google authentication on purpose. Those were decisions. I weighed them. They are healthy cessions because I made them.

Then there is the elephant in the room: layer 2C. Google has a fairly complete 2C offering — the reasoning and placement layer that decides where compute goes based on storage requirements, policy, and the rest. As an application builder I do not want to make those routing decisions by hand. I want to write organizational policy and opinions and let the platform place the workload. The problem is that when I built the application, I had no idea this layer existed. This was me learning how to use AI. So I did not just cede this authority to Google — I ceded authority I did not know I had to give. That is the real drift, and it is exactly the dependency that came to light when I tried to move to the Spark. This layer is where 4+1 came from.

What the model told me, and what it did not

The model was good at the thing it was built for. It identified the drift and predicted where that drift would bite if I moved the application — to AWS, to Azure, to on-premises, to the Spark. Point the assessment at the app, hold it against what each destination actually offers, and the cessions you never decided show up as the work you would have to do.

Where it reached its edge is day-two operations. The model says nothing about the authority drift in running the application. How do I do CI/CD if I am on Cloud Build today — how does that translate to AWS, or to Azure? What does logging look like? Security alerts? The control plane, if I want to grant rights to this application to other developers and administrators? None of that is in the 4+1 read, and the migration is what made the silence obvious.

That edge is not a flaw to paper over. It is the boundary of what the model is for. The 4+1 model identifies the components you need to build an application. It does not identify the authority you need to assign to run one. That is a different domain, and that work is for the greater community to do. Finding the edge of your own framework by stressing it against something real is supposed to be one of the outcomes. This is one.

The part worth sitting with

Drift is a bill you have not been handed yet. Everything works today, so the ceded authority costs nothing — until something forces it to resolve. A migration is one thing that forces it. So is a feature you cannot build on a captive layer, a scale you cannot reach, a requirement you cannot meet. The bill comes due eventually. The mock migration just handed me an early copy.

I went looking because I built a scenario where someone asked me to move. What I found was that almost everything in the application was ceded, not decided — and the two things I did decide, the model runtime and identity, I can point to and defend. The rest drifted in, one convenient default at a time, starting with a single choice to put the platform estate on Firestore. I did not mean to cede the pipeline, or 2C, or the retrieval strategy. I meant to cede the model and the identity. Now I know the difference.

Knowing the difference is the whole point of the read. It does not tell you to avoid lock-in. It tells you which locks you backed into by accident, so you can decide which ones you actually meant.

That sets up the next move. This piece was detection — finding the ceded authority and naming what drifted versus what I decided. In part two I go back through the same application and place the authority on purpose: take each cession the assessment surfaced and decide it deliberately, the version of this work I wish I had done before I ever wrote a line of the Virtual CTO Advisor.

I run a lab. I’m a practitioner analyst, I put this stuff on real hardware and watch where it breaks. The pattern is consistent: no single vendor delivers a complete enterprise AI stack. You compose one. Completeness is built, not bought.

So I did two things about it.

First, I graded the vendors. Layer by layer.

A real AI stack isn’t one product. It’s a set of layers, and most buyers can’t see the seams. So I made the seams explicit. I call it the 4+1 model:

• Compute and networking at the bottom.
• The data plane above it: storage, retrieval, pipelines.
• Orchestration and runtime in the middle.
• The reasoning plane near the top. This is the layer that decides what runs where based on business context, not just cluster metrics.
• It’s the real differentiator, and it’s the layer most enterprise stacks are quietly missing.
• The application layer on top, where the business value shows up. That’s the “+1.”

Then I graded the major vendors at every layer. Each grade carries two reads, not one. Can the vendor actually do the job at that layer? And who keeps the decision authority: you, or the vendor? Capability tells you the layer works. Authority tells you what judgment you’re borrowing from your platform vendor without noticing. Both live at layer2c.com, and every grade is something I tested, not a feature checklist.

Then I built Stack Builder, so you can compose with those grades.

Stack Builder is live at stackbuilder.layer2c.com. It’s free. No login, no limit.

You tell it what you’re trying to do, or you start from an easy-button: public, private, or hybrid. It composes a complete 4+1 architecture across vendors, layer by layer, from the graded assessments. Not from a model guessing what vendors do. From grades I stand behind. When it places a vendor at a layer, you can click straight through to the assessment that backs it.

Then it does the part the booth never does. It shows you the trade.

Every composed stack comes with two signals. The first is integration effort: where stitching best-of-breed across vendors breaks the native integration a single platform builds for itself. The second is reasoning dependency: how much autonomous judgment the stack is built to run, and therefore how much it matters who owns that reasoning plane. A single public cloud scores low on integration effort and high on reasoning dependency. Easy to assemble, and you’ve handed the hardest decisions to one vendor. That’s the trade most buyers back into.

You can edit the whole thing. Drag a vendor onto a layer and the stack recomposes. Don’t like the reasoning plane the easy-button picked? Swap it and watch the signals move.

What it is, and what it isn’t.

Stack Builder is an educational instrument, not a turnkey generator. It won’t hand you a finished design to go build unadvised. There is no turnkey AI stack, and the tool says so out loud. The easy-buttons are illustrative patterns. They teach the shapes and what each shape costs you.

What it’s good at is making the invisible choice visible. The most expensive mistake I see isn’t picking the wrong vendor. It’s ceding your reasoning plane by default, then discovering the placement in a post-incident review instead of choosing it before you deploy. Neither placement is wrong. Unplaced authority is wrong.

So here’s who should open it. If you’re about to standardize on one platform because the demo was clean, spend ten minutes composing the alternative and read the two signals. If you’re building on-prem and think a strong storage layer means you’re covered, look at where your reasoning plane actually sits. Then make the call on purpose.

Create an account and it saves your roadmaps. That’s the only thing login gets you. Everything else is open.

Go try it: stackbuilder.layer2c.com. Tell me where it’s wrong. The grades are tested. The way it stitches them together earns its authority in the field, by people using it, and that starts now.

The question that actually matters now is not whether the enterprise has infrastructure. It is whether the enterprise has a control model. Where should a given workload run? Which data is it allowed to touch? Which policies apply? What context can cross a boundary? And when cost, latency, compliance, and capacity all point in different directions — who, or what, has the authority to decide?

That last word is the one I care about most. Authority. Not capability. Authority is the question I have spent years trying to make legible, and it is the reason I built cloud.layer2c.com.

What the map actually measures

The new site is a companion to layer2c.com. The parent site explains the 4+1 AI Infrastructure Model and the role of Layer 2C, the reasoning plane. The companion applies that model to real platforms against a different test: the Fourth Cloud operating model.

Fourth Cloud is not a product category any vendor sells today. It is an operating pattern — a uniform, AI-native infrastructure model spanning cloud, on-prem, and edge, with governance and control integrated across the environment. The first public map scores VMware Cloud Foundation, Red Hat OpenShift, Nutanix Cloud Platform, and Oxide Computer. It will grow.

It is not a magic quadrant and not a vendor ranking. Every platform is scored on two axes that most evaluations collapse into one. The first is capability: function by function, on a 0–4 gradient, with AWS as the benchmark 4. The second is the one I built this whole body of work around — the Decision Authority Placement Model, or DAPM. For each function, the question is not “how good is it” but “can I take the opinions I have accumulated here and operate them somewhere else?” Three answers: Retained (yes — commodity or open-source, I can leave without rebuilding), Delegated (partly — a substitutable partner or swappable open-source tool), and Ceded (no — a closed system, and leaving means rebuilding).

Completeness and authority are independent. A platform can be the most complete thing in the assessment and the most captive at the same time. Hold that thought, because it is the headline finding.

The authority finding

Here is the DAPM profile across all four platforms, counting the twenty-six scored functions each:

Read that table next to the capability scores and the tension jumps out. The two platforms with the broadest coverage of the Fourth Cloud function set — VCF and Nutanix — are also the two most captive. Roughly two-thirds of what you build on either platform is Ceded: the configurations, the policies, the operational model are the vendor’s, and you cannot lift them to a competitor without rebuilding. That is not a criticism. It is the trade you are actually signing for, and it is usually invisible at purchase time because the demo is about capability, not authority.

A note on what “broadest coverage” means here, because it is narrow. This assessment measures completeness against the Fourth Cloud function set — how many of the FC-0 through FC-4 functions a platform addresses and how well. It is not a claim about general platform maturity, engineering quality, or enterprise track record. OpenShift, for one, would reasonably argue it is the more mature platform by most general measures, and that argument is not in scope here. What the table shows is coverage shape against one specific model, scored on two axes — nothing more.

OpenShift sits in the middle, and the reason is structural. Its open-source heritage shows up directly in the authority column — Kubernetes, Keycloak, the operator ecosystem, the GitOps tooling are Retained or Delegated, not Ceded. You pay for that portability with a higher assembly and operations burden, but the opinions are more yours to keep.

Oxide is the surprise only if you were reading the capability column alone. It is the most authority-preserving platform in the map — 85% Retained — because it is built on a co-designed, largely open substrate and does not try to capture the layers above it. Which brings me to the assessment I think matters most.

Each platform tells a different part of the story

VMware Cloud Foundation has the broadest coverage of the Fourth Cloud function set in this group. It is strong at the substrate and at orchestration, with genuine VM, container, and AI workload management through one console — and VCF 9.1 now manages AMD, NVIDIA, and Intel accelerators, which makes it the broadest multi-accelerator platform in the set. The weaknesses are equally clear: the distributed data fabric is thin, the integration fabric is effectively absent, and the reasoning plane does not exist. The authority profile says the rest — the widest coverage here, and one of the two most Ceded.

Red Hat OpenShift has a different shape. Its strength sits higher in the stack — execution, application distribution, integration, and a genuinely federated identity plane through Keycloak that spans more layers than any other platform here. It is also the clearest illustration of the line between rule dispatch and reasoning. ACM placement policies, node affinity, taints and tolerations, OPA/Gatekeeper — these are powerful primitives. They do not become a reasoning plane until the platform can derive a placement decision from live governance metadata without an operator translating every new constraint into a rule by hand.

Nutanix Cloud Platform brings a strong HCI operating model and the broadest workload management of the on-prem group — VMs, Kubernetes, databases, AI inference, and cloud burst through NC2 — under one Prism Central surface. It is a serious answer for enterprises looking to leave VMware. And its authority profile is nearly identical to VMware’s: a complete control plane that is mostly Ceded. The integration fabric is absent and the reasoning plane is absent.

Oxide Computer is the most important assessment for a reason that has nothing to do with its scores. Oxide is not trying to be a complete Fourth Cloud control plane. It is a deeply integrated IaaS substrate, and the honest comparison is EC2 or GCP Compute Engine, not OpenShift or VCF. That is not a weakness — it is a clean layer boundary. Oxide may well be the substrate a Fourth Cloud control plane runs on. The instrument’s job is to make that boundary visible instead of letting a vendor’s marketing blur it, and the authority numbers confirm the architecture: Oxide leaves the enterprise holding the most of its own opinions.

The finding nobody closes

No platform in the assessment closes FC-2C, the reasoning plane.

That is not a cheap shot — it is the entire point of the exercise. FC-2C is the layer that decides where work runs based on policy, compliance, cost, latency, data gravity, and capacity, simultaneously, and derives that decision from live metadata rather than from a rule an operator wrote last quarter. It is not the model thinking harder. It is not a smarter dashboard. It is the enterprise control point that decides whether work is operating within authority.

When FC-2C is absent, the decision logic does not disappear. It lives somewhere else — in an architecture review board, in Terraform modules, in Kubernetes admission policy, in a ServiceNow workflow, in a platform engineer’s head. The danger is believing the platform has absorbed that responsibility when it has only automated pieces of the workflow around it. That gap — the capability you quietly inherit while assuming the vendor owns it — is where most private cloud programs have failed. The technology worked. The operating model did not.

What this map does not measure

There is a limitation worth stating plainly, because it is the first thing a practitioner will notice: the instrument scores each platform alone, and almost nobody deploys these platforms alone.

The pattern I see in the field is a pairing. A substrate and orchestration platform — Nutanix, VCF, or Oxide — mated to an application platform, almost always OpenShift, layered on top. That is not an accident, and the map’s own data predicts it. The substrate platforms are strong low in the stack and absent at the integration fabric. OpenShift is the inverse: weaker at the substrate, strong at execution, application distribution, and the only real FC-4 integration fabric in the set. They are complementary by construction. An enterprise stacking OpenShift on Nutanix or VCF or Oxide is assembling the coverage that no single platform delivers.

The instrument does not yet score that combination, and the reason is not laziness — it is that a combination cannot be scored by adding two rows. Three things break. The orchestration layer becomes contested, because both platforms have one and the combined score depends on how cleanly they federate at the seam rather than on either platform alone. The authority profile compounds rather than averages: running OpenShift on VCF means you are now Ceded to both vendors at different layers, and the real question becomes who owns the seam between them when it breaks. And the identity plane rarely survives the handoff — OpenShift’s federated identity, its strongest authority asset, sits on top of a substrate identity model that usually does not federate upward, so the combined identity continuity is often worse than OpenShift’s alone.

One finding is worth previewing, because it inverts the single-platform read. On capability scores alone, Oxide looks like the least complete platform in the map. But Oxide is 85% Retained — it preserves more enterprise authority than anything else here. Pair OpenShift’s higher-stack coverage with Oxide’s authority-preserving substrate and you get the lowest combined Ceded surface of any pairing: the most authority-preserving way to assemble a near-complete Fourth Cloud on-prem. Nobody is having that conversation, because Oxide gets dismissed on its single-platform capability column before anyone looks at the authority column. The combination view is where that changes, and it is what I am building next.

Why this is paired with a readiness framework

The map ships alongside the Fourth Cloud Readiness Assessment and Evaluation Framework, and that pairing is deliberate. Fourth Cloud is not only a vendor problem — it is an operating-model problem. Can your organization run infrastructure as a product? Can you sustain gap ownership across vendors, coordinating API changes, roadmaps, integration lifecycles, and compliance policy over a span of years? Can you tell the difference between a capability the vendor owns, a capability you configure, and a capability you quietly inherit?

That last category is the expensive one. The framework puts a number on it — a single owned gap runs to roughly $1.5–2M over five years once you account for the build, the maintenance through vendor upgrades, and the lifecycle coordination nobody staffs for. The map shows you where the gaps are and who holds the authority. The framework helps you decide whether you can afford to own them.

What it changes

When you stop asking “which vendor wins” and start asking the two-axis question — what can this platform do, and what authority do I keep — the conversation in the room changes. You stop buying a feature list and start mapping an operating model. You can see which decisions the platform makes for you, which ones you still own, where policy becomes enforcement, and where automation quietly becomes authority you didn’t realize you were ceding.

The Fourth Cloud is not something you buy whole today. It is something enterprises are trying to assemble. The only question worth asking before the purchase order is whether you understand what you are assembling, what you still own, and where authority actually lives.

cloud.layer2c.com is live. The map will grow, the assessments will evolve as vendors respond and buyers ask sharper questions, and the model will keep improving. But the two-axis view is the part I would not give up: capability tells you what a platform can do, and DAPM tells you who is in control when it does it.

Let’s start with the unquestionable advantages of AI-assisted code. Boris Cherny, the creator and head of Anthropic’s Claude Code, says he hasn’t written a line of code by hand in 2026. He ships dozens of pull requests a day from his phone, relying on a system of agents to handle everything from code creation to PR reviews. Company-wide, Anthropic reports that AI now writes the overwhelming majority of its code. This is not a vendor projection. It is a working engineer describing how he builds today. Inside large organizations, the same acceleration is visible in a smaller way: application developers are quietly resisting calls to curb their token burn, because the tools genuinely make them faster and they don’t want to give that up.

So the technology works. Developers are accelerating. And yet the returns aren’t showing up on the bottom line.

Cherny, the person automating his own job out of existence, says the real gap is organizational, not technical. The man who has most thoroughly solved code generation is telling you that code generation was never the constraint.

The data backs him up. Bain’s Automation and AI Pathfinder Survey 2026, with 951 respondents, found that nearly 40% of the companies that actually measured their AI cost savings came in below 10%, against a target of 11% to 20%. Bain’s own conclusion was blunt: the technology worked, the value didn’t arrive.

In retrospect, this is not an unexpected outcome. Faster code generation doesn’t produce revenue. It produces inventory — features sitting half-finished in a review queue, the software equivalent of parts stacked in front of a station that can’t keep up. Manufacturers have understood for a century that work-in-progress isn’t value; it’s cost waiting to be realized, and only realized if the rest of the line can carry it. A factory produces a commodity, but that commodity has to be housed, inventoried, ordered, packaged, and transported before anyone realizes revenue from it. Traditional manufacturing has spent a century building strategies to optimize that order-to-cash pipeline for its specific business: some run just-in-time, others are rewarded for holding significant inventory. The strategies differ. The lesson doesn’t. Speeding up the line does not, by itself, strengthen the bottom line.

The illusion AI creates is that it removes the friction of getting product to the consumer. In reality, what AI-assisted code removes is friction at one station: production. It increases the velocity at which requirements become code. That only matters if there is a pipeline ready to carry that faster output the rest of the way. If your business isn’t built to absorb faster production, you have optimized a single point in the process without improving the throughput of the whole.

Look at what actually sits downstream of code generation in a mature enterprise. A feature an agent drafts in an afternoon still has to pass code review, clear integration, survive QA, get security and compliance sign-off, wait for a change-approval window, and finally ride a release train that may only leave the station every two weeks. The agent made the first station three times faster, or produced three times the code. It did nothing to the eleven stations behind it. So the work-in-progress piles up in front of the review queue, exactly the way half-finished parts pile up in front of a slow inspection station on a real factory floor. You haven’t moved the constraint. You’ve just made it more expensive to feed.

This is why the developers defending their token burn are, in a sense, defending the one station that already got faster, while the bottleneck sits untouched three stations down. The token meter is the cost of running that one machine harder. It tells you nothing about whether the finished feature ever reaches a user any sooner.

So why do some companies see a real return? Bain’s data answers this directly, and the answer is not what most IT leaders want to hear. The companies that hit their targets didn’t find better technology or secure bigger budgets. They treated data access, governance, and process redesign as CEO-level problems rather than IT problems. The ones that missed cite organizational obstacles — no center of excellence, competing priorities, insufficient mandate. Those aren’t technology failures. They’re signals that the business never re-engineered the pipeline to collect what the faster line was producing.

Companies born in AI, or willing to rebuild their pipeline around faster production, capture the gain because there is no eleven-station gauntlet between the line and the consumer. Everyone else bolts a faster machine onto an unchanged factory and is surprised when the output pools in front of the same old bottleneck.

None of this is new. It’s Theory of Constraints wearing an AI costume. The practitioner’s job hasn’t changed: find the actual constraint, and stop optimizing the station that was never the problem. AI didn’t fail to deliver ROI on code. The organization declined to change the factory that would have let it.

You do not have a migration theory problem. You have a migration capacity and authority problem.

Most enterprises already know the broad mechanics of moving applications from private data centers to public cloud platforms. The harder problem is execution at scale. There are not enough developers to rewrite every application manually. There are not enough infrastructure and platform engineers with deep development knowledge to inspect every workload. Systems integrators face the same constraint when they inherit incomplete inventories, uneven test coverage, unavailable application SMEs, and inconsistent runbooks.

LLMs can help, but only if they are used inside the right operating model.

This is not an AI migration factory. It is a migration control plane that uses AI only where developer-like adaptation is required.

In this model, the playbook drives execution, deterministic code performs known transformations, validators define completion, humans handle authority boundaries and exceptions, and LLMs are called only for bounded tasks that require interpretation, code adaptation, or remediation.

The core loop is simple:

LLM proposes. Playbooks constrain. Deterministic code enforces. Landing zone validates. Traceability records authority.

This model is based on three foundations. The first is the Decision Authority Placement Model (DAPM): authority must be placed deliberately across the migration process. The second is practical migration experience: 28 years of data center, infrastructure, and cloud migration work show that migrations fail as much from operating model mismatch as from code or infrastructure defects. The third is Google’s TensorFlow-to-JAX migration pattern, which demonstrates that large-scale AI-assisted migration becomes credible when probabilistic agents operate inside deterministic planning, playbook, validation, and evidence boundaries.

As enterprises experiment with agent factories and code-migrating LLMs, this paper provides the control-plane and authority model required to let those agents safely touch production-adjacent migration work.

The practical question is not whether an enterprise can use LLMs in migration. The practical question is:

What authority can we safely give the system today?

If the organization does not have enough rigor to manage playbooks as governed artifacts, adding an LLM will not create that rigor. It will only automate the absence of it.

Who This Paper Is For

This paper is for enterprise cloud, platform, and migration leaders — and the systems integrators who support them — who already understand cloud migration but need a safer way to scale repeatable migration work with LLM assistance.

The primary reader is responsible for moving a large application estate without unlimited developers, perfect application knowledge, or uniform test coverage. That includes cloud platform leaders, enterprise architects, infrastructure leaders, migration factory owners, application modernization leads, and senior cloud transformation teams.

The secondary reader is the systems integrator or cloud professional services team helping clients move from migration labor to migration operating model transformation. For these teams, the opportunity is not to provide more migration hands. It is to help enterprises build the migration control plane that makes scarce hands scale.

This paper is also for governance, security, and risk leaders who need to understand where authority lives, what gets validated, and how failures are traced back into the system.

This paper is not for teams trying to learn basic cloud migration. It is for organizations asking a harder question: how do we safely automate repeatable migration patterns across a large estate without giving agents uncontrolled authority?

1. From AI Migration Factory to Migration Control Plane

The goal is not to remove humans from cloud migration. The goal is to reserve scarce human judgment for the decisions that actually require it.

A traditional migration model often assumes that experienced humans will inspect the application, understand the code, identify platform gaps, rewrite what is needed, validate behavior, interpret failures, and decide when the application is ready for the target environment. That model breaks when the application estate is large, the development knowledge is uneven, and the migration timeline is compressed.

The migration engineer often knows what needs to change operationally but may not have the development depth to safely inspect every code modification by hand. That is already true with human developers. The migration engineer identifies the task, delegates the implementation, and validates the result using runbooks, tests, deployment checks, and operational evidence.

An LLM-assisted process should follow the same concept. The migration engineer does not need to personally prove every line of code is correct by inspection. The engineer needs a controlled process that can verify whether the change satisfies the migration requirement.

The migration process should not be described as LLM-driven. It is playbook-driven.

The playbook is the controlling artifact. It defines the sequence of steps, deterministic actions, validation gates, evidence requirements, and escalation rules. The LLM is called only when the playbook reaches a task that requires interpretation, code adaptation, ambiguity handling, or developer-like remediation.

The LLM should be treated the way a senior migration engineer would treat a developer: a callable capability for specific work, not the owner of the migration. The engineer may ask for help refactoring a configuration loader, interpreting a failed test, adapting framework-specific startup behavior, or proposing a candidate patch. The engineer does not ask the developer to own the entire migration process.

The same boundary applies to the LLM. The playbook owns the process. The scheduler owns routing. Deterministic validators own completion. Humans own exceptions and authority changes. The LLM owns only the bounded task it was called to perform.

Agentic loops become useful when the exit condition is outside the agent.

The Control Plane

A cloud migration loop becomes trustworthy when the loop has machine-enforceable playbooks and deterministic exit conditions. The agent is not trusted because it is smart. The agent is useful because it can search the solution space. The system is trustworthy because external validators determine whether the proposed migration is acceptable.

The migration control plane includes:

• Scheduler / intake inspection
• Playbook catalog
• Validator catalog
• Landing-zone profile
• Evidence store
• Exception process
• Human review boundaries
• RCA feedback loop

These components decide what the loop is allowed to do, when the loop is allowed to stop, and how the system learns when the loop was wrong.

Authority is distributed across the system:

Scheduler: classify and route
Playbook: define the approved execution path
Deterministic code: execute known transformations
Validator: accept or reject completion
Landing zone: accept or reject deployment fit
Human reviewer: approve exceptions and unresolved ambiguity
RCA loop: improve the system after failure

A simple control-plane flow looks like this:

Application Intake
→ Scheduler
→ Playbook Catalog
→ Deterministic Actions
→ LLM-as-Needed Tasks
→ Validators
→ Evidence Store
→ Human Exceptions
→ RCA Feedback Loop
→ Updated Scheduler / Playbooks / Validators

This paper is not a primer on how to perform cloud migrations. It is a primer on how to automate cloud migrations at scale once the organization already has enough migration discipline to describe, govern, and validate repeatable work.

Agent Harness vs. Migration Control Plane

As the market matures, some of this discussion will show up under the language of an agent harness. That is useful language, but it does not fully replace the control-plane model.

An agent harness gives the model execution capability. It may provide tools, memory, context, permissions, state management, observability, approval flows, and feedback loops. In other words, the harness helps the agent act.

The migration control plane answers a different question: when is that action allowed to matter?

In enterprise migration, the control plane must still decide which workloads are eligible for automation, which playbooks apply, what confidence level the scheduler has, what validators define completion, what evidence is retained, and where humans retain exception authority.

Agent harness governs execution. The migration control plane governs authority.

This distinction matters because a strong harness can make agents more useful, observable, and controllable, but it can also become the place where decision authority silently accumulates. If the harness allows the agent to assess, plan, transform, validate, and learn from feedback without explicit authority boundaries, the enterprise has not solved the trust problem. It has simply moved the trust problem into the harness.

The point of the migration control plane is to prevent that drift. The agent can act only inside the authority granted by readiness level, scheduler confidence, playbook lifecycle, deterministic validators, and human review boundaries.

2. Readiness Model: What Authority Can the System Safely Hold?

Playbook-driven LLM-assisted migration assumes the organization can already describe how certain classes of workloads should move, what the target landing zone requires, and how success will be validated.

Readiness should not be treated as a binary yes/no. Most organizations will be ready for some parts of the model before they are ready for governed automation. The practical question is not “Can we use LLMs in migration?” The better question is: what authority can we safely give the system today?

Level 1: Not Ready for Automation

At this level, the organization lacks the operating foundation for automated migration. Common indicators include unreliable application inventory, unclear application ownership, incomplete or inconsistent CMDB data, no stable landing-zone standard, limited test coverage, few repeatable migration runbooks, informally known policy requirements, relationship-based exception handling, no consistent evidence retention, and no RCA feedback process.

LLMs can help with application documentation cleanup, interview summarization, migration assessment support, dependency discovery assistance, drafting candidate runbooks for human review, and creating first-pass application summaries. They should not be used for automated code transformation, scheduler-driven playbook assignment, autonomous remediation, or production-impacting migration decisions.

At this level, adding an LLM to execution will accelerate ambiguity. The priority is building migration operating discipline.

Level 2: Ready for Assisted Assessment

At this level, the organization has some migration discipline, but not enough governed playbook maturity to grant transformation authority. Application inventory exists but needs cleanup. Landing-zone standards exist but are not fully machine-enforceable. Some migration runbooks exist, and some application patterns repeat.

LLMs can help with pattern discovery, application assessment reports, dependency and configuration summaries, draft playbook generation, gap analysis against landing-zone standards, assessment-only scheduler mode, and human-supervised pilot support. Broad automated transformation, unsupervised playbook execution, automatic exception remediation, and production promotion based on LLM-assisted changes alone are not yet appropriate.

This is where many enterprises should begin. The system can gather evidence, classify likely patterns, and help humans mature repeatable playbooks. It should not yet own transformation authority.

Level 3: Ready for Constrained Automation

At this level, the organization has enough governed artifacts to automate narrow migration paths under defined authority boundaries. Application inventory is reliable enough for scheduler input. Landing-zone capabilities are defined and inspectable. Approved playbooks exist for specific workload classes. Deterministic detection and validation rules exist. Evidence capture is standard. Human review boundaries and exception workflows are explicit.

LLMs can perform scoped developer-like remediation inside approved playbooks, test failure interpretation, candidate patch generation, and configuration refactoring when deterministic code cannot safely complete the step. Human review remains required for exceptions, low-confidence cases, and production-impacting changes.

This is the first level where playbook-driven LLM-assisted migration becomes operationally useful beyond assessment.

Level 4: Ready for Governed Automation at Scale

At this level, the organization can run a migration factory with playbook-driven automation across known workload classes. The approved playbook catalog exists. The validator catalog exists with owners and versions. Scheduler decisions are traceable and replayable. The landing-zone profile is machine-readable and current. Evidence store is integrated into migration governance. Playbook lifecycle is enforced. Human approvals and overrides are traceable. The RCA feedback loop updates the control plane.

At this level, the organization is not trusting the LLM to migrate applications. It is trusting the migration control plane to decide when, where, and how the LLM may be used.

Adoption Path

The practical adoption path should be incremental:

Assess applications
→ identify repeatable patterns
→ draft playbooks
→ pilot deterministic validators
→ run supervised LLM-assisted remediation
→ approve narrow playbooks
→ expand automation authority only with evidence
→ feed failures back through RCA

The maturity question for each organization is:

What can we deterministically inspect?
What can we deterministically transform?
What can we deterministically validate?
Where do we need LLM-assisted developer-like work?
Where must human authority remain in the loop?

Automation amplifies the operating model you already have. It does not create one for you.

How Authority Flows Through the Control Plane

The authority model has four related layers.

These layers prevent authority from silently accumulating in the LLM. A mature organization may be ready for governed automation, but a low-confidence workload still drops to assessment-only mode. A scheduler may identify a known pattern, but a draft playbook cannot drive automated execution. A playbook may execute successfully, but validators determine whether the work is done.

3. Scheduler / Intake Inspection

The first inspection at the scheduler layer is: does the prospect application conform to a known migration pattern?

This is the first routing decision. Before the agent attempts migration, the scheduler must determine whether the application matches a known pattern, partially matches a known pattern, or falls outside the current playbook library.

This inspection is not merely descriptive. It determines which playbooks are eligible, which validators must run, whether the migration can proceed automatically, and whether human review is required before any transformation begins.

Scheduler Responsibilities

The scheduler should inspect the application and classify it before assigning work to an agent loop. Initial scheduler responsibilities include:

• Identifying application type and runtime pattern
• Detecting framework, language, build system, and packaging model
• Identifying deployment model, such as VM, container, Kubernetes, batch job, or serverless candidate
• Detecting stateful dependencies
• Detecting identity and secrets patterns
• Detecting network assumptions
• Detecting storage and persistence model
• Detecting observability and logging patterns
• Detecting external service integrations
• Determining whether the app conforms to a known migration archetype
• Selecting the appropriate playbook or playbook chain
• Escalating unknown or ambiguous patterns for human classification

Scheduler Determinism

The scheduler should be deterministic wherever it is making an authority-bearing routing decision.

That does not mean every input must be perfect or every classification must be certain. It means the scheduler’s decision process must be repeatable, inspectable, and based on declared rules rather than hidden model judgment.

A scheduler may use an LLM to summarize an application, extract candidate facts, or explain findings. But the scheduler should not rely on the LLM as the final classifier for playbook assignment. The routing decision should be made by deterministic rules over inspectable evidence.

LLM assists inspection.
Deterministic scheduler classifies.
Human reviews ambiguity.

The scheduler’s job is not to understand the application like a human architect. Its job is to determine whether enough evidence exists to route the application into a known playbook, a constrained pilot path, assessment-only mode, or human review.

Scheduler Inputs

The scheduler should classify an application using concrete inspection artifacts:

• Repository metadata
• Dependency manifests
• Build files and runtime version files
• Dockerfiles or container manifests
• Kubernetes manifests
• Terraform or other IaC
• CI/CD pipeline definitions
• Application configuration files
• Static analysis and secrets scanning results
• Network dependency scan
• Runtime inventory or CMDB data
• Observability and logging configuration
• Existing test coverage
• Application owner questionnaire, if needed
• Landing-zone capability inventory

Pattern Classification

The scheduler’s first useful output is a pattern classification.

Known Pattern. The application conforms cleanly to an existing migration pattern. For example, a stateless Java Spring Boot application running on a VM with an external database, standard HTTP ingress, and environment-based configuration may be a candidate for containerization and deployment into an approved runtime. The scheduler assigns the known migration playbook, runs deterministic preflight checks, and allows the migration loop to begin transformation within approved scope.

Known Pattern with Exceptions. The application mostly conforms to a known pattern but has specific exceptions. For example, a stateless web app may store secrets in local files, use local-only logging, or depend on a hardcoded internal hostname. The scheduler assigns the primary migration playbook, attaches exception-specific remediation playbooks, requires additional validation gates, and may require human review before deployment.

Unknown Pattern. The application does not conform to any known pattern in the migration library. Examples include custom runtimes, undocumented binary dependencies, shared filesystem assumptions, unclear state models, and undocumented integration paths. The scheduler does not begin automated transformation. It generates an inspection report, routes to a human architect or migration engineer, and may create a candidate pattern for future playbook development if the pattern recurs.

Prohibited Pattern. The application matches a known pattern that is not approved for automated migration. Examples include regulated workloads requiring manual security review, unsupported runtimes, legacy apps with direct hardware dependencies, applications with unclear data classification, and workloads requiring unavailable landing-zone capabilities. The scheduler stops the loop, produces an evidence report, and escalates to governance or architecture review.

Degraded-Confidence Scheduler Mode

The scheduler should not be written as if classification always resolves cleanly. In practice, this is one of the first places the model will break. Application evidence will be incomplete. Repository structure will be inconsistent. CMDB data may be stale. Owners may misunderstand their own dependencies. Static analysis may detect symptoms without understanding architectural intent.

A deterministic scheduler still needs a degraded-confidence operating mode. Degraded confidence does not mean the scheduler failed. It means the scheduler produced enough evidence to continue the assessment, but not enough evidence to grant full automation authority.

The scheduler should support several confidence states:

High confidence: Known pattern, full required evidence present, no disqualifying rules triggered.
Medium confidence: Known pattern with exceptions, evidence mostly complete, warnings require remediation playbooks or human confirmation.
Low confidence: Possible pattern match, but required evidence is missing or contradictory.
No confidence: Unknown pattern or insufficient evidence to classify.
Prohibited: Known disqualifying rule triggered.

The important point is that confidence changes the authority granted to the downstream process.

High Confidence / Automated Candidate. The application is eligible for approved playbook execution. The migration loop may run within approved scope. Human review is not required until an exception, failed validator, or production release gate appears.

Medium Confidence / Constrained Automation. The application is eligible for supervised or constrained playbook execution. The migration loop may run detection, remediation, and test-environment transformation only. Human review is required before production-impacting changes or deployment promotion.

Low Confidence / Assessment-Only Mode. The system should not transform yet. Agents or deterministic tools may inspect, summarize, and gather additional evidence. Human review is required before any code or infrastructure changes.

No Confidence / Human Classification Required. The automated migration path stops. No transformation authority is granted. Human review is required to classify the pattern or create a candidate draft playbook.

Prohibited / Governance Stop. The process stops. No automation authority is granted. Human review is required only for exception or policy review.

A deterministic scheduler does not have to pretend it knows. It has to be explicit about what it knows, what it does not know, and what authority that uncertainty permits.

Scheduler Traceability and Inspectability

Every scheduler decision must generate a trace record. At minimum, that record should include the scheduler run ID, application ID, migration request ID, scheduler version, rule set version, playbook catalog version, landing-zone profile version, input artifacts inspected, evidence extracted, rules evaluated, classification result, confidence state, playbooks considered, playbooks selected or rejected, automation authority granted, additional evidence requested, escalation reason, human reviewer if applicable, and timestamp.

The trace record should make it possible to replay or inspect the decision later. The scheduler is not just plumbing. It determines what kind of automation is allowed to touch the application.

4. Playbooks as Governed Automation Artifacts

There are multiple playbooks in a cloud migration. They should not be treated as one giant migration prompt. Each playbook should encode a specific pattern, transformation, or validation concern.

Candidate playbook categories include:

• Application pattern playbooks: stateless web application migration, stateful application migration, batch job migration, API service migration, event-driven application migration, legacy monolith modernization, containerization candidates, serverless candidates, and Kubernetes candidates.
• Platform conformance playbooks: identity and access management, secrets management, logging and observability, network ingress and egress, DNS and certificates, storage and persistence, backup and recovery, tagging and cost allocation, encryption and key management, deployment pipeline integration, runtime configuration, and policy-as-code conformance.
• Remediation playbooks: hardcoded secrets remediation, local file dependency remediation, hardcoded hostname remediation, local logging remediation, database connection modernization, runtime version upgrades, dependency replacement, configuration externalization, and IAM or service account migration.
• Validation playbooks: unit test validation, integration test validation, behavioral equivalence validation, infrastructure-as-code validation, security policy validation, runtime smoke testing, observability validation, performance baseline validation, and rollback validation.

A playbook is stronger than a markdown file when it is machine-enforceable through deterministic code. A markdown instruction can describe the desired state. A governed playbook defines detection logic, required transformation, validation logic, exit condition, evidence produced, and escalation rules.

Example structure:

Playbook ID: secrets-management-gcp-001

Policy Source:
All cloud workloads must use approved managed secrets storage.

Detect:
- .env files committed to repo
- Hardcoded credential-like strings
- Kubernetes Secret manifests in app repo
- Local config files containing secret-like values

Transform:
- Move secrets to approved secrets manager
- Update app bootstrap code
- Update IAM/service account permissions
- Update deployment config

Validate:
- No secret-like values remain in repo
- Secret manager resource exists in IaC
- Workload identity can access required secret
- App starts successfully without local secret file
- Policy scanner passes

Exit Condition:
All validation checks pass.

Evidence:
- Rule ID
- Files inspected
- Findings
- Changes applied
- Test results
- Policy result
- Reviewer or exception owner
- Timestamp

5. Playbook Execution: Deterministic First, LLM as Needed

Each playbook step should first ask: can this action be performed deterministically?

If the answer is yes, the playbook should use deterministic code rather than the LLM. Deterministic code is faster, cheaper, more repeatable, easier to test, and easier to audit.

Examples include detecting committed .env files, identifying secret-like values using scanners, creating approved secret manager resources, updating environment variable references in deployment manifests, rewriting known configuration keys, updating Terraform modules, adding required tags or labels, validating IAM bindings, checking whether logs reach the approved sink, and running unit tests, smoke tests, policy checks, and deployment checks.

LLM-as-Needed Execution

The LLM should be called when the playbook reaches a task that cannot be safely handled by deterministic code alone. Examples include interpreting unfamiliar application logic, refactoring custom configuration loading code, adapting framework-specific code to a new runtime contract, explaining why a test failed after migration, proposing a code change when deterministic remediation cannot infer the right transformation, mapping undocumented behavior to a known migration pattern, and generating candidate patches for human or deterministic review.

The LLM call should be scoped to a specific task, bounded by the playbook, and followed by deterministic validation.

Step: Refactor configuration bootstrap
Deterministic pre-check: local config file dependency detected
LLM task: propose code patch to read non-secret config from environment variables and secrets from approved secret accessor
Constraints: do not change business logic; do not introduce new dependency outside approved list; preserve existing config key names where possible
Validation: unit tests pass; app starts without local config file; secret retrieval integration test passes
Exit condition: deterministic validation passes

The LLM may perform developer-like tasks, but the acceptance model remains the same: the change must pass the defined validation gates.

Validation Without Code Expertise

The migration engineer may not be able to inspect the code deeply enough to know whether the implementation is elegant, idiomatic, or maintainable. But they can validate whether the migration requirement has been satisfied.

Requirement: Application no longer reads secrets from local files.
Validation:
- no local secret file reads detected
- secret manager access path exists
- service account has required permission
- app starts successfully without local secret file
- integration test retrieves secret at runtime
- policy scanner passes

The migration engineer is not validating the code by taste. They are validating the operational outcome.

Review Boundary

Some changes still require developer or architect review. Examples include business logic changes, data model changes, concurrency changes, authentication or authorization logic changes, error-handling behavior changes, performance-sensitive code paths, code changes that pass tests but alter user-visible behavior, and repeated LLM remediation attempts that do not converge.

The playbook should distinguish between changes that can be accepted through deterministic validation and changes that require expert review.

Playbook Execution Trace

Because the playbook is the driver, the execution trace should record each step as a playbook action, not as a free-form agent conversation. Minimum playbook execution fields include playbook execution ID, application ID, playbook ID and version, step ID, step type, execution mode, inputs used, artifacts changed, deterministic tool or script version, LLM model and prompt if used, human reviewer if used, validation result, evidence produced, retry count, exit condition status, and timestamp.

This makes the migration inspectable as a controlled process rather than a transcript of agent behavior.

6. Case Study Walkthrough: Stateless Java Application with Secrets and Logging Exceptions

Scenario

A migration factory is moving a VM-hosted Java Spring Boot application from a private data center to an approved public cloud landing zone.

The application has:

• External database dependency
• HTTP ingress
• Local configuration files
• Local file-based logging
• Credentials stored in application.properties

The target landing-zone pattern requires:

• Containerized deployment to an approved runtime
• Managed identity
• Secrets stored in the approved secret manager
• Structured logs emitted to the approved logging path
• Required tags and ownership metadata
• Deployment through an approved CI/CD pipeline
• Policy-as-code checks before promotion

Walkthrough

The LLM task in this walkthrough is deliberately narrow:

Task: Refactor configuration bootstrap
Context: The app currently reads secrets and environment configuration from local application.properties.
Constraint: Do not change business logic.
Constraint: Preserve existing config key names where possible.
Constraint: Use approved secret accessor and managed identity.
Constraint: Do not introduce unapproved dependencies.
Output: Candidate patch only.
Validation: deterministic tests and policy gates decide acceptance.

The validators check both old-world behavior and new-world landing-zone conformance.

Old tests prove the app still works: the build succeeds, unit tests pass, existing API smoke tests pass, and database integration tests pass.

New tests prove the app belongs in the landing zone: no secret-like values remain in the repo, secret manager resources exist, runtime identity can retrieve secrets, the app starts without a local secret file, logs appear in the approved logging sink, required metadata and tags exist, policy-as-code checks pass, the container image builds, and the test deployment succeeds.

Old tests prove the app still works. New tests prove the app belongs here.

Why the RCA Matters

The missed secret-manager outage failure is not a side case. It is the reason the migration control plane needs closed-loop learning. The app passed the validators that existed at the time. The RCA process determines whether the miss was workload-specific, organization-specific, or general enough to update the reusable migration system. Section 8 expands that feedback loop into an operating model.

7. What Breaks: Failure Modes and Validator Gaps

The model is strong, but only if playbooks are treated as governed artifacts, not clever prompts.

Bad Pattern Classification. The scheduler may classify an application as a known pattern when it only superficially resembles one. The application may look stateless while depending on local disk persistence, shared filesystem locks, or implicit network trust. Mitigation requires confidence thresholds, human review for low-confidence matches, and negative detection rules.

Playbook Drift. A playbook may reflect yesterday’s landing-zone pattern, not today’s approved platform standard. The loop then enforces outdated policy deterministically. Mitigation requires versioned playbooks, reviewed changes, execution records, and fail-closed deprecated playbooks.

Over-Automation of Exceptions. The system may treat recurring exceptions as normal and automatically remediate them without architectural review. The migration factory hides real design problems by repeatedly patching symptoms. Mitigation requires exception frequency tracking, playbook review triggers, and restrictions on auto-remediation for certain exception classes.

Deterministic but Wrong. A deterministic check may pass while enforcing the wrong requirement. The organization gains false confidence because the pipeline is green. Mitigation requires rules that trace to approved policy, rule owners, adversarial and edge-case test examples, and evidence that shows why each rule exists.

Validator Coverage Gap. A more subtle failure occurs when the LLM proposes a change that passes every deterministic validator but is architecturally wrong in a way the validators were not designed to catch. This is not an LLM-only problem. Human developers can also make changes that pass tests while violating architectural intent. The problem is that the validator set becomes the effective definition of correctness. If the validators are incomplete, the loop can confidently accept the wrong outcome.

A green pipeline proves the workload satisfied the validators. It does not prove the validators captured every architectural concern.

Landing-Zone Assumption Gap. The playbook may assume the target platform has capabilities that are not actually available or enabled. The agent generates a valid target design that cannot be deployed in the actual enterprise landing zone. Mitigation requires scheduler inspection of landing-zone capabilities before playbook assignment, declared platform capability requirements in playbooks, and hard stops for missing capabilities.

8. Validator Governance and Closed-Loop Learning

The secret-manager outage example in Section 6 is the practical reason validator governance matters. The system did not fail because it used an LLM. It failed because the validator set did not capture a real architectural risk.

Because validators define the exit condition, they are authority-bearing control plane components. The system needs a process for governing the validator set itself.

If the validator defines done, the validator is part of the authority model.

The validator governance process should answer:

• Who owns each validator?
• What policy, runbook, or architectural requirement does it enforce?
• What failure mode is it intended to catch?
• What failure modes is it not intended to catch?
• How was the validator tested?
• What known gaps exist?
• What evidence does the validator produce?
• What happens when the validator passes but downstream review finds an architectural problem?

RCA Feedback Loop

Validator gaps are inevitable. This is true for human-written software, human-led migrations, and LLM-assisted migrations. A workload can pass every known validator and still fail because the validators did not capture the relevant failure mode.

The answer is not to pretend validation can become perfect. The answer is to make failures traceable, perform root-cause analysis, and feed the learning back into the scheduler, playbooks, validators, landing-zone profiles, and organizational knowledge base.

The RCA process should begin whenever a migrated workload fails after passing the expected validation gates. The purpose of RCA is not only to fix the workload. The purpose is to improve the migration control plane.

Every post-validation failure should create an RCA trace that links the failure back to the migration execution history: application ID, migration request ID, failure event, failure description, impact, detection source, playbook version, scheduler run ID, validator set and versions used, LLM calls involved if any, human approvals or overrides involved if any, evidence available at migration time, evidence missing at migration time, root cause category, corrective action, preventive action, feedback destination, owner, due date, and closure evidence.

RCA Root Cause Categories

RCA should classify failures into categories that lead to different fixes.

• Missing validator: the failure mode was not covered by any validator.
• Weak validator: a validator existed but did not test the condition deeply enough.
• Bad scheduler classification: the application was routed to the wrong playbook or given too much automation authority.
• Playbook assumption gap: the playbook assumed a migration pattern that was not true for this workload.
• Landing-zone profile gap: the target environment did not provide the capability or operating behavior the playbook assumed.
• Organization-specific control gap: the migration passed general cloud validation but violated a local standard, operational convention, or compliance requirement.
• Human decision error: a human approved an exception, override, or classification that later proved wrong.
• LLM remediation error: the LLM produced a change that passed available validators but introduced a hidden defect.

General vs. Organization-Specific vs. Workload-Specific Learning

Not every RCA should update the global playbook. Some findings are generally reusable. Others are specific to the workload, application family, business unit, or organization.

General learning applies across many migrations and should improve the reusable control plane: validator catalog, playbook catalog, scheduler rules, landing-zone profile, pilot test suite, and migration engineering runbooks.

Organization-specific learning reflects local standards, platform design, compliance rules, or operating practices. It should update the organization policy map, landing-zone profile, local validators, local evidence requirements, and local exception workflows.

Workload-specific learning applies to one application or tightly related application family. It should update the application migration record, application-specific runbook, CMDB/application portfolio metadata, future scheduler evidence for related workloads, and human review notes.

The feedback process should not pollute global playbooks with one-off exceptions, but it should also avoid burying reusable failure modes inside application-specific notes.

Closed-Loop Improvement

The migration system should not simply record RCAs. It should require closure before similar migrations continue at the same automation level.

Minor workload-specific issue: update application runbook.
Recurring pattern issue: update playbook and validator set.
High-severity control failure: suspend playbook automation and return to pilot.
Policy violation: stop affected migration class until governance review completes.

Passing validation is not the end of governance. It is the beginning of evidence-based learning when reality disagrees.

The question is not whether failures will happen. They will. The question is whether the system can tell what failed, why it failed, and which part of the migration control plane must improve.

9. Human-in-the-Loop as Accountability, Not Correctness

Human review does not eliminate error. A human can misclassify an application, approve a bad exception, misunderstand the landing-zone requirement, or rubber-stamp an agent’s recommendation. The purpose of human involvement is not to make the process infallible. The purpose is to place accountability, judgment, and exception authority where the organization can inspect it.

Human review provides accountability. It does not automatically provide correctness.

Human decisions need traceability for the same reason LLM decisions need traceability. The system must show what evidence was available, what decision was made, who made it, what authority they had, and what downstream automation that decision enabled.

An LLM can produce a decision-like output without accountable ownership. It can summarize evidence, infer patterns, and recommend a playbook, but it does not carry organizational authority. Its output must be treated as evidence or recommendation unless bounded by deterministic rules.

A human can carry organizational authority, but that does not mean the human is correct. The human can approve the wrong thing. The difference is that a human decision can be assigned to a role, reviewed against policy, challenged later, and used to improve the process.

Human review should be triggered when the system reaches an authority boundary: unknown patterns, low-confidence classifications, disqualifying rule override requests, production-impacting exceptions, missing landing-zone capabilities, policy conflicts, repeated failed loop attempts, and migration paths with material cost, risk, or architecture implications.

The human should not merely approve the agent’s output. The human should approve a specific decision with evidence.

Weak review:

Looks good. Proceed.

Strong review:

Approved as Known Pattern with Exceptions.
Reviewed scheduler evidence showing Java Spring Boot app, external database, no local persistent state, and two warnings for local secrets and file logging.
Approved remediation playbooks secrets-remediation-gcp-001 and local-logging-remediation-gcp-001.
No production deployment allowed until validation gates pass.

Human-in-the-loop is not a correctness guarantee. It is an accountability boundary.

10. Playbook Lifecycle: From Human Practice to Governed Automation

This section is not intended to teach the basics of migration planning. It assumes the organization already has migration practitioners, landing-zone standards, and enough operating discipline to recognize repeatable patterns.

The purpose of playbook maturation is to convert known migration practice into governed automation. If the organization cannot already describe how a class of workloads should move, it is not ready to automate that class of migration.

A playbook should begin as a human-governed migration pattern, not as an AI-generated artifact.

Observed migration pattern
→ human architect classification
→ target-state decision
→ policy and landing-zone mapping
→ deterministic detection rules
→ deterministic validation rules
→ pilot migrations
→ evidence review
→ approved reusable playbook

A playbook should move through controlled lifecycle stages:

Draft
→ Pilot
→ Approved
→ Deprecated
→ Retired

Each stage requires explicit promotion criteria. A playbook should not move from draft to pilot or pilot to approved because the team feels confident. It should move because it has produced enough evidence to justify the next level of authority.

Draft. A draft playbook is a human-authored description of a recurring migration pattern and the desired target-state pattern. It may be assisted by AI, but it is not yet allowed to control an automated migration loop.

Pilot. A pilot playbook is allowed to run against selected applications under human supervision. It can guide a migration loop, but it cannot make unsupervised production-impacting decisions. The pilot should include a clean match, a partial match with expected remediation, a false-positive risk, and a disqualification or escalation case.

Approved. An approved playbook is eligible for scheduler assignment and migration loops within its approved scope. It is still not allowed to expand its own scope, approve its own exceptions, ignore failed validators, change policy source mappings without review, or update landing-zone assumptions without recertification.

Deprecated and Retired. Deprecated playbooks remain traceable for old migrations but are not assigned to new ones. Retired playbooks are no longer valid, though historical execution evidence remains available for audit.

Concrete Lifecycle Example: Local Secrets Migration Playbook

Migration engineers repeatedly find applications reading credentials from .env files, application.properties, or local configuration files on VMs. The target landing-zone standard requires all secrets to be stored in the approved cloud secrets manager and accessed through managed workload identity.

In draft, the playbook defines eligible apps, disqualified apps, target state, detection logic, and validation rules. It is advisory only.

In pilot, the team selects four applications: one clean Spring Boot app using application.properties, one Node.js app with a .env file and straightforward environment variable mapping, one app with a suspicious local config pattern that might be a false positive, and one app with a custom credential broker that should be rejected.

The pilot shows that the deterministic scanner correctly identifies common secret patterns and rejects the custom credential broker case. It also reveals a gap: some non-secret environment values were being classified as secrets. The classification logic is updated and the pilot is rerun.

After evidence review, the playbook is approved for applications within a narrow scope: common framework-based local secret patterns, test and staging transformation, and production promotion through normal release controls. Custom credential brokers remain excluded, and regulated workloads require separate governance review.

Six months later, the landing-zone team introduces a new secrets access library and deprecates the old accessor pattern. The previous playbook remains traceable for past migrations but is no longer assigned to new applications. After all remaining applications have moved to the new pattern, the old playbook is retired. Historical execution evidence remains available for audit.

11. Traceability and Evidence Chain

Once playbooks are machine-enforceable, they become part of the audit surface. Each playbook must be traceable from business requirement to enforcement result.

Business requirement
→ migration policy
→ playbook rule
→ deterministic enforcement code
→ test result
→ deployment decision
→ exception record, if any

The goal is to prevent the agent from hiding policy decisions inside code changes.

Every playbook execution should produce an evidence chain. Minimum evidence fields include application ID, migration request ID, pattern classification, playbook ID and version, policy source, detection result, transformation applied, validation result, artifacts changed, deterministic checks executed, human reviewer if required, exception owner if applicable, timestamp, and final disposition.

The agent should not be allowed to hide policy decisions inside code changes.

12. The SI Role: More Work, Higher-Value Work

For a systems integrator, this model does not reduce the work. It changes the work.

The traditional migration story often treats scale as a staffing problem: assign more people, run more assessment waves, execute more runbooks, and push more applications through the factory. That model still runs into the same constraints the client faces. The SI does not magically inherit perfect application knowledge. The SI often faces the same blockers: incomplete inventories, inconsistent runbooks, uneven test coverage, unclear ownership, landing-zone drift, unavailable application SMEs, and limited developer capacity.

The migration control plane does not eliminate those blockers. It exposes them, structures them, and creates a path to reduce them over time.

The SI opportunity is therefore not smaller. It is larger and more strategic. The work moves upstream from migration labor into migration operating model transformation.

Instead of only moving applications,
the SI helps build the system that makes application movement repeatable, inspectable, governable, and improvable.

The SI does not win by pretending the mess is gone. The SI wins by turning the mess into a control plane.

SI Work Across the Maturity Curve

The readiness model creates a natural SI engagement path. This section uses the maturity curve from the SI delivery lens: what work the SI performs at each stage. Section 2 uses the same curve from the enterprise buyer lens: what authority the organization can safely grant the system at each stage.

The overlap is intentional. The SI view describes the engagement model. The readiness view describes the operating authority model.

Level 1: Build Migration Discipline. For clients not ready for automation, the SI helps create the operating foundation: application inventory cleanup, ownership mapping, dependency discovery, landing-zone readiness assessment, current-state documentation, migration wave planning, runbook normalization, test coverage assessment, and policy and exception process discovery.

Level 2: Assisted Assessment and Playbook Discovery. For clients ready for assisted assessment, the SI helps discover and document repeatable patterns: application pattern discovery, candidate playbook drafting, assessment-only scheduler design, landing-zone gap analysis, human review workflow design, policy-to-control mapping, initial evidence model design, and pilot workload selection.

Level 3: Constrained Automation. For clients ready for narrow automation, the SI builds and operates the first governed playbook loops: deterministic detector development, validator catalog development, playbook implementation, CI/CD and policy-as-code integration, test-environment migration loops, LLM task constraint design, evidence capture implementation, human approval workflow integration, and pilot RCA and playbook refinement.

Level 4: Governed Automation at Scale. For mature clients, the SI can help operate the migration control plane across the application estate: migration control plane operations, playbook catalog lifecycle management, validator catalog maintenance, scheduler rule governance, landing-zone profile updates, exception workflow management, RCA facilitation and closure, organization-specific validator development, workload-specific knowledge capture, reporting and evidence management, and continuous improvement of automation authority.

Why This Is Higher-Value Work

The lower-value migration work is the repetitive labor that can eventually be encoded, validated, and repeated: finding common secrets patterns, updating known configuration references, rewriting boilerplate deployment manifests, applying standard tags, checking known policy requirements, and producing repetitive migration documentation.

The higher-value work is defining and operating the system that makes those tasks safe to automate: deciding which patterns are known enough to automate, building playbooks that encode approved target states, creating deterministic validators, defining human review boundaries, managing exceptions, interpreting failures, feeding RCA back into the control plane, and separating global learning from organization-specific and workload-specific learning.

The work does not disappear. It moves upstream into the design and operation of the migration control plane.

SI Business Implication

This model can be uncomfortable for SIs that depend primarily on migration labor volume. But it is attractive for SIs that want to sell higher-value transformation.

The SI can package the work as a progression:

Migration readiness assessment
→ application pattern discovery
→ playbook factory design
→ validator catalog development
→ constrained automation pilot
→ migration control plane operations
→ closed-loop optimization

The result is not fewer SI services. It is a different mix of services: more architecture, platform engineering, governance, automation, testing, evidence management, and operational transformation.

The SI opportunity is not to provide more migration hands. It is to help the enterprise build the migration control plane that makes scarce hands scale.

13. Open Questions

The following questions remain open and are intended to guide future work, implementation planning, and organizational adoption decisions.

• What is the minimum application inspection data required before assigning a playbook?
• Should the scheduler classify by application architecture, deployment target, operational risk, or all three?
• What qualifies as enough confidence to allow the loop to begin?
• Which failures should retry automatically versus escalate immediately?
• How should new unknown patterns be promoted into reusable playbooks?
• Who owns playbook approval: platform team, security, architecture, app owner, or shared governance board?
• How much of the playbook should be represented as deterministic code versus human-readable documentation?
• How often should approved playbooks be recertified against the current landing zone? For example, many organizations would align this with landing-zone release cycles or perform at least quarterly review.
• What is the threshold for converting recurring exceptions into new remediation playbooks?
• What validator gaps require suspension of an approved playbook versus a minor version update? For example, gaps that can cause data integrity loss, security policy violations, or production availability risk should usually trigger suspension until reviewed.
• How should organization-specific learning be separated from global playbook improvements?
• Where should workload-specific knowledge live so future scheduler runs can use it without polluting global playbooks? For example, application portfolio metadata, CMDB records, or application-specific migration runbooks may be better destinations than the global playbook catalog.

The first is the Decision Authority Placement Model — DAPM. It identifies drift in decision-making and authority. We have seen it repeatedly: automated systems begin to quietly overtake operational authority without explicit authority being granted. DAPM gives organizations a way to see that drift before it becomes structural.

The second is a domain-specific application of the theory: the 4+1 AI Infrastructure Framework. It maps the AI stack into eight layers that matter to enterprise architecture — compute and network fabric, data storage and governance, context and retrieval, data movement, orchestration, runtime execution, the reasoning plane, and the application layer — and applies DAPM at every layer. I recognized the infrastructure pattern before I recognized the general theory. The 4+1 model came first. DAPM emerged from asking why the same authority questions kept appearing across every vendor assessment.

Today I am publishing the instrument that ties them together: layer2c.com — an independent, public assessment of 12 AI infrastructure vendors that shows how enterprises cede, delegate, or retain authority to platform providers when adopting their technology stack.

The problem

Companies are struggling to identify value from AI. The models are good enough. The GPUs and software are good enough. The problem is not the technology.

My anecdotal experience is a reasonable proxy for what is possible. I have built a seven-figure run rate with just me, my wife, and a few thousand dollars a month in AI spend. I have used patterns I have seen across my 28-year career to capitalize on AI as a co-thinker. Over the past year I have built the following:

• Virtual CTO Advisor
• AI Stackbuilder
• 4+1 AI Infrastructure Framework + RFP
• AI Factory Economic Model
• Decision Authority Placement Model
• The Fourth Cloud Readiness Assessment
• Layer2C.com
• Analyzed 15 years of Tech Field Day content
• Produced several whitepapers, an ebook, and video content

Pre-AI, you could pick any two of those projects, and I would have been happy to accomplish both in the same year. Conservatively, I produced two years of output in less than one. I no longer have the skill to develop code, yet I built three platforms running in the public cloud — another low-level skill I no longer claim.

The productivity gains are real. But as I like to say: scale breaks everything.

I got into a fascinating debate with Jon Stevens, founder of NeoCloud Hot Aisle. Jon’s claim is that in a few short years, AI will displace storage software complexity, and companies like Dell will only sell NFS mounts that scale to petabytes. His evidence is a peer who manages 20PB of storage with a collection of scripts.

The logic is the same as my productivity hack. It is easy for small teams to realize productivity gains from automation and AI. The challenge comes when you try to scale it across distributed teams.

Can I use AI to create a block-level replication scheme? Yes. Can I create a data pipeline that auto-populates a vector store? Sure. Can I operationalize it to manage 20PB of data? Maybe. Can I have a team of 30 across four sites support it through a generation or two of employee turnover? Highly unlikely.

This is the repeated pattern for automation and AI projects. It is relatively easy for born-in-AI companies and small teams to realize the productivity gains. But scale breaks everything.

As a small business, I can identify the gap, perform rapid triage, select a solution, implement the technology, monitor for drift, and iterate — all inside my own head. Large enterprises are much more complex. They need systems for scaling the selection, management of technology, and controlling drift.

Enter Layer 2C

Layer 2C is the reasoning plane — the layer where policy, placement, escalation, evidence, and decision authority become explicit. It is where AI infrastructure stops being a collection of independently managed capabilities and starts becoming a governed operating model.

That layer is where most vendor stories break down. Not because the technology is weak, but because the question changes. It is no longer “Can the system run the workload?” It is “Who determines what the system is allowed to do, how it reasons, where it executes, and how that decision is inspected?”

That is the question layer2c.com is built to answer — vendor by vendor, layer by layer.

Capability is not authority

The simplest way to describe what the site does is this: capability coverage and authority placement are not the same thing.

A vendor may have broad coverage across the AI stack and still require the enterprise to place significant decision authority inside that vendor’s control boundary. Another vendor may preserve more enterprise control but leave gaps in higher-level orchestration, runtime, or reasoning-plane capability. Neither answer is automatically good or bad. They describe different control bargains.

DAPM classifies where that authority sits, using four positions:

Retained means the enterprise preserves a decision surface that can survive vendor substitution. The organization may have to rebuild implementation details, but it does not have to re-decide the architecture.

Delegated means a substitutable partner provides the capability. The enterprise depends on someone else for implementation, but its architectural requirements can survive a partner change.

Ceded means the enterprise has expressed its architecture in vendor-native constructs that do not easily survive a platform move. Substitution requires architectural re-decision, not just operational rebuilding.

Absent means the capability is not meaningfully present at that layer. Note: This is not canonical for DAPM. It’s added for this context. 

A ceded capability can be excellent. A retained capability can be immature. Delegation can be smart. Absence can be intentional. DAPM is not a quality score. It is a way to describe where authority lives and what the enterprise would have to re-decide if the platform changed.

What the site does

Layer2C publishes vendor assessments against the 4+1 Layer AI Infrastructure Model. Each assessment includes a layer-by-layer view, component-level notes, gap analysis, borrowed-judgment commentary, and a DAPM authority profile.

Twelve vendors are live today: Dell, HPE, AWS, VAST, Google Cloud, VMware by Broadcom, Microsoft Azure, NVIDIA, IBM/Red Hat, Oracle Cloud, Palantir, and Cisco. They span infrastructure OEMs, hyperscalers, data platforms, virtualization layers, component providers, open-source platform plays, networking and security vendors, and application-layer platforms that build downward from the decision rather than upward from silicon. 

What the assessments found

The framework produced findings that no single vendor’s assessment would reveal on its own. These are initial findings from the first assessment set — vendor stacks evolve, and assessments will update as capabilities ship. But a few patterns are worth naming here, because they demonstrate what the instrument is designed to surface.

Layer 2C is the defining gap — and the defining differentiator. Dell is absent at the reasoning plane. VMware is absent. NVIDIA does not build it. HPE is the only on-prem vendor with a productized Layer 2C — GreenLake Intelligence for IT operations, Kamiwaza via Unleash AI for AI workloads — and that two-part story is structurally unique. VAST has the most ambitious Layer 2C architecture, a closed governance-and-learning loop across PolicyEngine, Polaris, and TuningEngine, but much of it ships later this year.

The cloud and platform vendors are approaching the reasoning plane from different directions. Google Cloud has the most complete productized Intelligence 2C. IBM makes the most explicit 2C claim — watsonx.governance and InstructLab are 100% IBM IP at a layer where every other IBM component is commodity Kubernetes. Microsoft is building 2C as an identity story through Entra Agent ID and the Agent Governance Toolkit. Oracle’s engineering team has published the most sophisticated thinking on 2C from any hyperscaler, but the gap between their published architecture and their shipped product is real.

Every vendor’s Layer 2C story is different. That is the point. The reasoning plane is where vendor strategies actually diverge, and the assessments make that divergence legible.

The NVIDIA dependency extends further than most enterprises realize. Every on-prem vendor depends on NVIDIA GPU silicon at Layer 0. That is expected. What the assessments make visible is how far that dependency reaches into the upper layers. At Layer 2A, NVIDIA provides GPU Operator, Run:ai, and scheduling primitives. At Layer 2B, NVIDIA provides NIM, NeMo, and inference frameworks. The enterprise choosing a Dell or HPE AI Factory is simultaneously choosing a significant portion of NVIDIA’s software stack. The NVIDIA dependency column in each assessment tracks this layer by layer.

The Dell-Palantir inversion is structurally complete. Dell owns Layer 0 and is absent at Layer 2C. Palantir is absent at Layer 0 and strongest at Layer 2C. Dell says “bring any Layer 2C” — and that Layer 2C arrives Delegated, substitutable, with authority distributed across partners. Palantir says “bring any Layer 0” — and everything above the floor consolidates into one Ceded-and-operated authority. Same invitation, opposite consequence. Their heat maps are literal inverses of each other, and the implications for how the enterprise’s control plane gets assembled are opposite as well.

Cloud vendors concentrate authority, not just capability. AWS, Google Cloud, Azure, and Oracle all provide strong or moderate capability at nearly every layer. The DAPM classifications tell the real story: the enterprise consuming these platforms cedes authority across the stack. That is often exactly why enterprises choose cloud platforms: operational leverage in exchange for governance concentration. But it is a trade-off the enterprise should make explicitly, not inherit by default.

These are not the only findings. Each assessment contains layer-level analysis that is specific to the vendor and its competitive position. But these four patterns illustrate what the instrument is designed to do: surface the structural reality underneath the capability story.

How to use the site

If you are evaluating or already running one of these vendor platforms, start with that vendor’s assessment. Read the summary finding for the structural thesis. Open Layer 2C to see the reasoning plane story. Check the DAPM classifications at the layers that matter most to your deployment — for most enterprises, that means Layers 1A (data governance), 2A (orchestration), and 2B (runtime). The borrowed judgment assessment at each layer tells you where your vendor depends on partners for decision-making authority that you will inherit.

If you are a vendor, start with your own assessment and read it as your enterprise buyers will. The gap analysis and borrowed judgment sections surface the questions that serious architects ask after the demo — where the platform preserves customer authority, where it depends on a partner’s roadmap, and where enterprise opinions get embedded in constructs that do not survive a platform change.

Then look at a second vendor. The framework is the same across all twelve, so the comparisons are structural. The cross-vendor comparison view aggregates the layer status and DAPM heat maps side by side.

What this is not

Layer2C is not a procurement recommendation engine. It will not tell a buyer to choose Dell over HPE, Google Cloud over Azure, VMware over AWS, or NVIDIA over the rest of the ecosystem. The right answer depends on the existing estate, the operating team, data gravity, application strategy, compliance posture, and how much authority the organization is willing to place inside a vendor’s control boundary.

Layer2C is also not a lab validation. It does not replace benchmarks, reference calls, proof-of-concepts, or hands-on technical evaluation.

It is decision intelligence. It gives buyers and vendors a structured way to ask better questions before the conversation collapses into feature comparison, roadmap promises, and partner-logo slides.

Why this matters

For vendors, the uncomfortable part of Layer2C is that it does not simply ask whether the product is good. It asks whether the story holds together as enterprise architecture. Where does the platform preserve customer authority? Where does it depend on a partner’s roadmap? Where is the reasoning plane explicit, and where is it implied?

These are the questions serious enterprise buyers eventually ask, even when they do not use this language. They may not say “Layer 2C.” They may not say “DAPM.” But they are asking whether the platform will support the operating model they have to run after the demo ends.

For buyers, Layer2C is a way to slow down the AI infrastructure conversation long enough to see what you are actually buying. The risk is not ceded authority — enterprises cede authority all the time. The risk is accidental ceded authority: signing up for a platform assumption without understanding which parts of the architecture you control, which parts you delegate, and which parts you have placed inside someone else’s roadmap.

If the buyer understands the bargain, it is architecture. If the buyer discovers it later during audit, migration, incident response, cost control, or governance review, it becomes technical debt.

Scale breaks everything. Layer2C is my attempt to show you where it breaks first.

You can explore the research at layer2c.com.

The reality is that the truth lies between these two marketing messages.

The problem is that AI spend for agentic coding is out of control. As of late May 2026, the time of this writing, organizations are reporting that they have exhausted all of their budget for agentic AI coding tokens. The simplistic solution would be to buy cheaper, more capable models from Google or to run your AI locally on Dell. My testing reveals something more nuanced.

We are back in the world of hybrid cloud, just for the AI revolution. However, we are 16 years into the dance between cloud providers and on-prem data center vendors. The industry has figured out the workload patterns and built abstractions and operating models that demarc between the workloads. We are early in AI and haven’t figured out the shape and size of the workloads while the capabilities advance faster than we can operationalize.

From the two events and my research, we can see the form of the operating model that keeps you ahead of the curve. It’s all about determining who decides done, so that you have the proper Layer 2C control plane for determining where to place your agentic workloads.

What Google Is Actually Selling

Gemini 3.5 Flash is not a cheap model. At $1.50 per million input tokens and $9.00 per million output, it’s 3x more expensive than the previous Flash. But it beats Gemini 3.1 Pro on coding and agentic benchmarks while running roughly 4x faster — around 284 tokens per second. Google is calling this frontier-class performance at half to a third the cost of OpenAI and Anthropic.

How can they do that? They built and control the whole stack. Their silicon, their data centers, their models. We’ve seen this play with AWS Lambda and Graviton. Except this looks much closer to what Apple does with their Apple Silicon to OS to Services integration.

Google services billions of users on the underlying Gemini model on TPUs across all of their services. While not exactly a rounding error, it’s an AI infrastructure cost structure advantage that doesn’t go away. It gets wider as inference demand scales. When your competitor is renting GPUs and you’re fabbing TPUs, you’re playing a different game.

Google is already building the cascade, too. At I/O they showed Gemma 4 on Cloud Run handling lightweight agentic work, escalating to Gemini 3.5 Flash for harder reasoning. Both in the cloud. Google’s golden path is a capability cascade within their own infrastructure from Android, to Cloud Run, to Gemini. They don’t need you to buy hardware. They need you to stay on their platform while they give you the cheap tier and the expensive tier under one roof.

What Dell Is Actually Selling

Dell’s economics case comes from a Signal65 study. Three agent personas — knowledge worker, sales agent, software development — running persistently 24 hours a day, 260 days a year. For software development on the GB300 Ultra, Signal65 found $1.06M in cloud cost versus $138K on-prem over two years. That’s up to 87% lower cost with payback in 3 to 11 months. The Dell Pro Max with GB10 — same chip as my DGX Spark — starts at $3,699.

I have to be clear here, this isn’t a realistic scenario. Anything with this type of demand would see a much more robust design that goes well beyond the single deskside failure domain. But the numbers are directionally right. Agentic token consumption compounds. A single developer at Dell burned through 1 billion tokens in 24 hours — $3,400 cloud bill. Multiply that across a team and cloud-only is not a strategy. It’s a budget surprise.

Dell’s golden path is also a cascade — start on a deskside workstation, scale to PowerEdge servers in the data center. Own the hardware, stop paying per token. Same thesis as Google, opposite side of the transaction.

What Both Miss

Both companies’ models fall into the same homogeneous trap that their marketing teams love. Google’s cascade stays within Google. Dell’s cascade stays within Dell. Neither models what happens in an already well-established pattern for enterprise workloads over the past 16 years of hybrid cloud. There will be boundaries based on workloads, and the judgment of where those boundaries fall is the work.

My experiment shows that matters. Last week I published “Who Determines Done?” — results from running real bug-fix commits from open source repos as coding tasks on my DGX Spark, with frontier escalation to o3 and gpt-5.5. Ten milestones, two workloads, six local model configurations. Total API cost for the entire project: $4.60 across 156 calls.

The difficulty distribution was bimodal. About 45% of tasks were trivially easy — local model, first attempt, done. About 50% were too hard for the local model no matter how many times you retry. The middle band where a repair loop actually helps? About 5%.

That thin middle band matters. The industry is building agentic architectures optimized for the 5% case and paying for it on 100% of tasks.

The three-tier escalation chain — local model gets one shot, o3 gets up to two, gpt-5.5 as the final fallback — took pass rate from 25% to 75%. The local tier cost nothing beyond hardware I already owned. The frontier tier was cheap because most tasks never got there.

Here’s the math that begins to show the patterns we’ve seen for the past 16 years, just adapted for hybrid AI. If ~45% of real coding tasks resolve locally at near-zero marginal cost and another ~25% resolve with a single frontier call at $0.04 to $0.24, the blended cost per finished job is dramatically lower than either the all-cloud or all-local number. You’re not comparing “$135K hardware versus $1M in cloud APIs.” You’re comparing “$135K hardware plus a few hundred dollars in targeted frontier calls versus $1M in cloud APIs where most of that spend is frontier-tier pricing on tasks a 30B model handles fine.”

Where Flash 3.5 Fits

Flash 3.5 changes the Tier 2 calculus. My experiment used o3 as Tier 2 and it worked — o3 is not the newest frontier model, it’s much cheaper than the latest ones, and it has a high success rate on coding tasks relative to cost. That’s the “Cost per Finished Job” argument. You don’t need the most capable model. You need the cheapest model that passes the exit gate.

Flash 3.5 at frontier-class capability and half to a third the cost? That could replace o3 as Tier 2 for a lot of tasks. I haven’t tested it yet. But the architecture is designed for exactly this — swap the model, keep the validator, measure the pass rate, compare the cost. The tier is a slot. The model is replaceable.

The Flywheel

The sweet spot right now is what I’m actually doing: running local models on a Spark for initial coding passes, then escalating to frontier models for the hard stuff. The local model runs at near-zero marginal cost after the hardware investment. The frontier calls are cheap because most work never gets there. The hardware pays for itself not because of some abstract TCO model, but because you stop paying per token for work a local model handles.

My math: the Spark pays for itself after roughly 28,000 requests that would have otherwise gone to a frontier API. That number gets better as local models improve. Six months ago I couldn’t run Gemma 4 MoE on this hardware. Now it handles 45% of real bug-fix tasks on the first attempt.

As local models get better, the percentage that resolves at Tier 1 goes up. The frontier spend goes down. The breakeven accelerates. That’s the flywheel neither keynote articulated.

Hybrid Is Not a Place

Google’s own cascade doesn’t run entirely in the cloud. I sat in a session where Google showed the cascade from an Android device to a laptop to Cloud Run using Gemma 4 to Gemini 3.5 Flash. Same architecture, same escalation pattern, different infrastructure. The decision about where to run is economic and governance, not architectural.

Hybrid is not a place. It’s an operating model. The same escalation architecture works whether your Tier 1 is a Spark in your lab or a Gemma 4 container on Cloud Run. It all returns to Layer 2C — the orchestration layer that decides which model, at which tier, gets the next attempt. That layer doesn’t care where the model runs. It cares about capability, cost, and the exit condition.

Who Determines Done

Stop defaulting to the most expensive model. Build a validator that determines “done.” Route to the cheapest tier that passes the validator. Escalate on failure.

The local tier handles more than you think — 45% of real-world coding tasks in my testing. The frontier tier is cheaper than you think — o3 at $0.04-$0.24 per task, not the $3,400/day horror story. The middle is where Flash 3.5 competes — frontier capability at structurally lower cost because Google owns the silicon.

The question across all of this is the same one I keep coming back to: who determines done? The validator determines done. The escalation policy determines what happens when done is not reached. The infrastructure underneath shifts the economics of both decisions. What counts as “done” changes as model capabilities change. But infrastructure matters because it determines what you can afford to attempt.

If you’ve been in enterprise IT long enough, you’ve seen this movie. Early cloud had the same fights. Build versus buy. On-prem versus public. Hybrid as a compromise versus hybrid as a real architecture. Took a decade to figure out the answer was not a location — it was a governance model. We’re watching the same thing play out with AI inference right now. The patterns aren’t set. Build the governance layer first. Let the infrastructure decisions follow.

Dell gave us the hardware economics. Google gave us the cheap cloud tier. My lab work points to the missing piece between them: a capability cascade with deterministic exit gates.

The full experiment, methodology, and raw data are here. The escalation pattern is L1/L2/L3 support — every enterprise architect already knows it. The validator is the authority. The loop is plumbing. The question is still who determines done.

That’s not either company’s story to tell. That’s what enterprise architects have to build.

The corrected finding is the more interesting one. For this class of problem, real bug fixes with an executable test, I have not found a need to escalate at all. The small local model, gated by the deterministic validator, clears the solvable work on its own. The frontier tier bought throughput, not correct answers. I re-ran the same instrument at rigor in a follow-on lab: a 12B model at 4-bit quantization cleared the solvable set with zero confirmed false passes across 585 attempts.

This does not weaken the argument below. It sharpens it. Looping a deterministic model is still useless, and the validator still determines done. What changed is who gets the credit. I gave it to escalation. It belonged to the small model and the gate.

-Original Post-

Last week I did something unusual. I published a requirements document for an experiment I had not run yet. The question was simple: where should judgment live in a local-first agentic AI system?

I ran the experiment. Ten milestones, two workloads, six local model configurations including Nemotron, Gemma, Qwen, and LLaMA variants, frontier escalation to o3 and gpt-5.5, all on a DGX Spark in my lab. Total API cost for the entire project: $4.60 across 156 calls, from the actual OpenAI billing export. Well, and the $4,000 I paid for the Spark :)

The answer was not what I expected. The result was not a better loop. It was a capability cascade with deterministic exit gates.

I tested several approaches for deciding when to trust the local model and when to escalate to a stronger model. The final design was a three-tier escalation chain that tries once, validates the result, and moves up only when needed. The first workload was RSS triage scored against my own editorial judgment. That exposed the failure modes but hit a ceiling — the answer key was subjective. The second workload was coding, where executable tests serve as the authority signal.

The coding workload had its own progression. I started with synthetic isolated tasks. Small, clean, well-specified. The local model passed everything on the first attempt. The loop never fired. The experiment was useless because the tasks were not hard enough to test loop control.

The real signal came from git archaeology. I extracted actual bug-fix commits from open source repos — httpx, more-itertools, click, requests, arrow, dateutil, prettytable, marshmallow, humanize. Real bugs that real developers filed PRs for. I rebuilt them as self-contained tasks with the original regression tests and feature tests. Real messy code, real regression risk, real complexity. That is where the loop architecture was actually tested.

BTW, how did I do all of this in four days? Claude Code. So, there is my disclaimer. This is directional. I did not go through every revision, check the code, and test to ensure accuracy.

Here is what I found.

The Industry Is Fixated on Loops

The agentic AI conversation right now is about loops. Tool-calling loops, ReAct loops, multi-agent loops, self-reflection loops. The assumption is that if you give a model tools and a loop, it will converge on the right answer. The loop is the innovation.

That assumption drove my experiment. I went looking for who should control the loop. Local model? Frontier model? Deterministic code? Some hybrid?

What I found is that the loop is plumbing. The interesting question is downstream of the loop.

Who Determines Done?

The question that actually matters is not who controls the loop. It is who determines done.

Deterministic code is surprisingly good at this. Tests pass or fail, schema validates or it does not, format checks, regression checks, diff-scope checks — these are machine-enforceable exit conditions. On my coding workload, deterministic validation accepted roughly half the tasks on the first local attempt across the broader extracted task set. In the harder subset I used for the final escalation comparison, the local pass rate was lower — 25% — because the easy tasks had already been filtered out during calibration. The model wrote code, the validator accepted it, done.

One important caveat: “done” only means as good as the authority signal. In coding, tests can define done with useful precision. In RSS triage, schema validity only proves the output is well-formed. It does not prove the routing decision is correct. That is why the architecture worked better once the workload moved from subjective editorial judgment to executable validation.

These are not synthetic difficulty labels. The tasks that pass trivially are real Python bugs from real PRs — a bytes in versus == check in h2, an iterator exhaustion bug in more-itertools, a type-check guard in PyJWT. Actual developers fixed these. The local model fixes them too, first try, every time. The tasks that fail at zero percent are also real — a parser state machine in httpx, a combinatorics algorithm in more-itertools, arithmetic reasoning in dateutil. Real developers struggled with these too.

The difficulty distribution turned out to be bimodal. Across 28 tasks extracted from 9 repos:

That thin medium band matters. It means the repair loop’s useful range is vanishingly small. Most tasks either resolve immediately or require a fundamentally more capable model. More on that in a moment.

The exception is quiet failures. Confident wrong answers that pass all structural checks. The model guesses, the validator sees nothing wrong, and the output looks clean. This is real. In the RSS triage workload, two items were routed incorrectly but passed every deterministic check — no schema failure, no proxy trigger, no signal for the control loop to act on. That limitation is genuine. But it is the minority case, not the common one.

The validator is the unsung hero of this architecture. Not the loop.

Looping Harder Does Not Help. It Can’t.

The industry assumption is straightforward: if the model gets it wrong, feed the error back and let it try again. The loop will converge.

My data says it does not. And on further testing, I found out it cannot — at least not at temperature zero.

Near-miss tasks proved it cleanly. One task passes 6 of 7 feature tests on the first attempt. So close. Feed the failing test output back, try again. Same result. Try again. Same result. Four attempts, identical output every time. Another task passes 4 of 5 tests on the first attempt. Four attempts, identical output every time. The local model produced functionally identical output across all four attempts on these near-miss tasks. The error feedback changed the prompt, but the model converged to the same fix every time — it was confident in the wrong approach, and nothing in the feedback pushed it to a different one. In practice, the repair loop converged to the same wrong answer.

Earlier milestones hinted at this more gently. The few tasks that did benefit from a repair loop converged on attempt two. Never attempt three or four. Those were cases where the model understood the fix conceptually but made an implementation error — a wrong character set in an RFC formula, a missing regression guard in a combinatorics function. One round of test feedback was enough. But that sweet spot turned out to be roughly 5% of tasks across 28 extractions from 9 repos. For everything else, the model either gets it right on the first attempt or it is stuck at a capability boundary that no amount of self-repair will cross.

Universal review made it worse. On the RSS triage workload, I tested a frontier model as an always-on governor reviewing every local output. Accuracy dropped from 8/10 to 6/10. I tested a local critique pass on every output. Also dropped to 6/10. Both exhibited the same failure modes. Review Drift: the reviewer challenges correct decisions, introducing new errors into previously clean outputs. Over-saving Bias: reviewers aggressively challenge “ignore” decisions but rarely challenge “save” decisions, promoting low-value items to avoid the risk of missing something.

I also tested whether the critique source matters. Self-feedback and external frontier critique produced identical outcomes on every task. The separation-of-duties thesis — that a model cannot judge its own work — was neither confirmed nor refuted. The critique source is not the variable. What matters is whether the model can act on the correction signal. At temperature zero, if it could not act on it the first time, it will not act on it the fourth time either.

Escalate, Do Not Loop

Once I understood that same-tier repair loops had a vanishingly narrow useful range at temperature zero, the architecture simplified dramatically. Do not loop the same model. Escalate to a more capable one.

I built what I called modeE: a three-tier escalation chain. Tier 1 is the local model — Gemma 4 MoE, self-hosted on the DGX Spark, free — and it gets one attempt because retrying a deterministic model is provably useless. Tier 2 is a frontier reasoning model — o3 via the OpenAI API — and it gets up to two attempts when test feedback is available, because o3 is non-deterministic and benefits from error steering. Tier 3 is the most capable model available — gpt-5.5 — as a final attempt only if needed. The deterministic test harness gates promotion between tiers. Each tier starts from a clean workspace with no context from prior tiers.

Correction: the 75% figure is superseded. Nine invalid validator checks had booked escalations for tasks that were actually unsolved on broken tests. Corrected, the escalation gain largely dissolves and the local tier carried the solvable work. See the update at the top.

The two-tier jump looked like it provided all the value. The later validator audit corrected that: much of the apparent escalation gain was invalid checks booking escalations for cases that were never solved. gpt-5.5 still solved nothing o3 failed, so the third tier earned nothing here either.

A later audit found this was not two tasks but nine invalid validators: empty or self-contradictory gates that no output could pass. The unsolvable-by-construction share was larger than I first reported, and it was a harness-quality problem, not a model ceiling. That only strengthens the next point.

That first failure points to a broader lesson. The validator becomes the authority, but that also means validator quality becomes part of the system’s correctness boundary. Deterministic code does not magically remove judgment. It moves judgment into tests, schemas, validators, and policy code. If the test is wrong, the architecture faithfully enforces the wrong answer.

Every enterprise architect already knows this pattern. It is L1/L2/L3 support. The help desk agent works from scripts. If the script does not resolve the ticket, the agent does not try the script harder. They escalate to L2. L2 has more capability and context. If L2 cannot close it, L3 gets it. Nobody calls that a loop. It is a tiered resolution architecture.

The validator plays the same role as the runbook in L1 support. It determines whether the issue is resolved at this tier. The escalation trigger replaces the “I cannot solve this” decision. The Decision Authority Placement Model is the formal version of what IT operations teams already do with human labor.

But What About Claude Code?

An obvious objection: Claude Code loops and it works extremely well. I used it to build this entire experiment in four days. If repair loops are dead, why does the most visible agentic coding tool on the market run one?

Because Claude Code’s loop is not a mechanical repair loop at temperature zero. It is a conversation. The developer reframes the problem, adds context, changes the prompt between attempts. The important distinction is that the state of the problem changes between attempts. Each iteration has genuinely different input, which means genuinely different output. That is not looping. That is a human providing escalation. The developer is L2.

I tested this directly. I gave o3 a second attempt on a task it failed, feeding back the test errors from the first attempt. o3 is non-deterministic — unlike the local model at temperature zero, it produces different output on the same prompt.

When o3 produces code but gets one test wrong, the error feedback consistently fixes it on the second attempt. Two of three runs recovered. The only failure was when o3 returned empty files — a different failure mode entirely where the model would not engage with the task.

The design rule is now empirically grounded on both sides. Deterministic workers — local models at temperature zero — get one attempt. Retrying is provably useless. Escalate immediately on failure. Non-deterministic workers — frontier models, or a human reframing the problem in Claude Code — benefit from a second attempt because each iteration is genuinely different. The error feedback steers the model somewhere new.

The repair loop did not fail because looping is wrong. It failed because looping a deterministic process is wrong. Match your retry policy to the worker’s determinism. The validator does not care either way. It just checks whether the output passes.

Tool Calling Should Be Treated Like L3 Work

Agentic tool calling is often the most expensive type of work in this architecture, and not just because of compute. Every tool call adds coordination overhead: parsing, execution, error handling, state management, permissions, observability, and another place where the system can fail. Complexity is a cost, even when the token bill looks cheap.

From the architecture’s perspective, agentic tool calling is irrelevant as a category. It is just one level of capability in the abstraction. The worker at any tier might use tools. A local model writing code is a worker. A complex agentic system with tool calling and RAG is a worker. The architecture does not care what happens inside the tier. It cares about the exit gate and the escalation policy.

But from a cost perspective, tool calling matters a lot. It is the most expensive tier. Running agentic tool-calling loops on tasks that a single-pass classifier or code generator would handle is paying L3 prices for L1 work.

A quarter of my coding tasks — real bugs from real repos — are solved by the local model on the first attempt. No tools, no loop, no agent, no cost. Running those tasks through an agentic architecture would add cost and latency for zero improvement in outcome. Another half are solved by adding a single frontier call at $0.04 to $0.24 per task. Only the genuinely hardest tasks need the most expensive tier, and even then, some are unsolvable by any model.

This connects to the cost-per-finished-job argument. The industry prices AI by the token. Genius per token. The capability cascade prices it by task resolution. The cheapest resolution is not merely the one with the lowest token cost. It is the simplest path that satisfies the exit condition: a single-pass model checked by deterministic code, when that is enough. Cost-per-finished-job includes compute, latency, operational complexity, failure surface, and governance burden. Tool-calling agentic systems are the expensive tier you invoke only when cheaper tiers cannot satisfy the exit condition. This is not anti-agent. It is anti-agent-by-default.

The purchasing decision is familiar. Nobody sends every support ticket to the engineering team. Nobody flies a consultant in for a password reset. Defaulting to the most expensive resolution tier for every AI task is the same mistake enterprises learned to stop making with human labor decades ago.

From the Outside, This Is How Thinking Models Already Work

Chain-of-thought. Internal verification. Escalation from fast System 1 to slow System 2. Spending more compute when the cheap pass does not satisfy the constraint.

From the outside, thinking models appear to follow a similar economic pattern. From the outside, they appear less like same-tier retry loops and more like systems that spend additional compute when the cheap pass is not enough.

The temperature-zero repair loop failure proves this from the negative side. A model that loops at the same capability level without escalating is doing exactly what my local repair loop did — producing the same wrong answer repeatedly. The observable value of thinking models is that they spend more compute when the cheap pass is not enough. My experiment rediscovered that principle externally, the hard way. And when I tested o3 with a second attempt, the non-deterministic frontier model behaved the way you would expect — it produced different output, and the error feedback steered it to the right answer.

The architecture I built is the same pattern, just distributed across hardware tiers with the validator as an explicit governable component instead of a hidden internal mechanism.

The difference that matters for enterprises: when the cascade is internal to the model, you cannot see the exit gate, you cannot audit the escalation decision, you cannot measure the cost at each tier, and you cannot place authority deliberately. Thinking models solved the capability cascade for themselves. Enterprises need to solve it for their agent architectures — and they need it to be visible, auditable, and governable.

What This Means for Layer 2C

Layer 2C in the 4+1 model — the reasoning plane — is not one model reasoning harder. It is the orchestration of a capability cascade with deterministic exit conditions at every tier.

The components are straightforward. A validator that determines doneness. An escalation policy that decides when to swap capability tiers. Workers at each tier that are interchangeable labor. Whether a worker is a local classifier, a cloud code generator, or a full agentic system with tool calling is an implementation detail scoped to that tier.

The pattern is fractal. Local model to cloud model to frontier reasoning model. Or 4o to o3 to 5.5. Or a local classifier to a local agentic system to a cloud agentic system. Same validator, same escalation policy, different capability at each level.

Authority placement follows the Decision Authority Placement Model. Authority over “is this done” lives in the validator. Authority over “escalate to a more capable tier” lives in the escalation policy. The model at each tier has authority over bounded work — nothing more.

The enterprises that understand this will resolve most tasks cheaply, on a small model behind a deterministic gate, and escalate only where a class of work actually needs it. For falsifiable coding work, mine did not.

Appendix – Data

Download the Raw Data and Lab Notes

The unusual part is not writing the requirements document first. The unusual part is publishing it before running the experiment.

Normally, the blog post comes after the experiment. You run the test, collect the results, draw the conclusion, and then write up what happened.

I have a DGX Spark in the lab. The interesting question is not whether it can run a model. Local inference is becoming more practical by the month. Quantization helps with model fit. KV cache helps with long-context execution. The hardware is good enough to make real experimentation possible.

The harder question is whether a local model can participate in a real agentic system when execution, validation, and judgment are separated deliberately.

That is the question behind this requirements document.

For the past year, I have been developing a few related ideas around enterprise AI infrastructure. My 4+1 AI Infrastructure Model argues that the missing layer in many enterprise AI conversations is Layer 2C — the reasoning plane. That is where orchestration, authority placement, escalation, and judgment live. The Decision Authority Placement Model makes a related argument from a governance angle: AI systems do not just fail because the model is weak. They fail because decision authority is poorly placed.

This experiment is a way to put those ideas under pressure.

The application is intentionally practical: RSS-to-opportunity triage. Given a new article, vendor post, research note, announcement, or transcript segment, the system has to decide whether the item should be ignored, saved, escalated for review, or turned into a draft opportunity note.

That is close to my actual editorial workflow. It is also easier to test than something subjective like “write in Keith’s voice.” The system either routes the item correctly or it does not. It either identifies the vendor, maps the topic, grounds the evidence, detects duplicates, and escalates uncertainty, or it does not.

The hypothesis is simple: local models may be excellent workers but unreliable governors.

A local model may summarize, extract, classify, and propose a decision. But the harder question is whether it should own the loop. Can it tell when the evidence is weak? Can it detect when it is summarizing instead of deciding? Can it know when to stop? Can it know when local judgment is not enough?

My guess is that the useful architecture is neither all-local nor all-cloud. It is local execution, deterministic validation, and selective escalation to a stronger reasoning model when the loop requires judgment.

In plain English: local workers, deterministic control, and outsourced judgment only when needed.

That is why I am publishing the requirements document before running the experiment. The control architecture is the content. Where the system places authority, what it validates with code, what it leaves to the model, and what triggers escalation are the same questions every enterprise will face when agent demos turn into production workflows.

The document below reads like a requirements document because it is one. That is intentional.

Everyone demos agents. The more useful question is who controls the loop.

Spark Loop Control Test: Requirements Document

1. Purpose

This document defines the requirements for testing loop-control architecture on the DGX Spark using OpenClaw as the consistent orchestration substrate across all test modes.

The experiment is designed to answer one core question:

Where should judgment live in a local-first agentic AI system?

The test is not intended to prove that a local model can run. It is intended to evaluate whether local models can serve as reliable workers when execution, deterministic validation, and frontier-model judgment are separated.

2. Core Hypothesis

Local models may be useful workers but unreliable governors.

The expected winning pattern is:

Local execution, deterministic validation, and selective escalation to a stronger reasoning model when the workflow requires judgment.

OpenClaw is the consistent orchestration substrate for Modes 0 through 4. This controls for platform behavior so that the experiment measures authority placement, not differences between agent frameworks.

3. Application Under Test

The application is RSS-to-opportunity triage.

Given a new article, vendor post, research note, product announcement, transcript segment, or RSS item, the system must decide whether the item should be:

• Ignored
• Saved
• Escalated for review
• Turned into a draft opportunity note

The system does not need to write in my voice. It needs to make a correct routing decision based on evidence, structured rules, and controlled escalation.

4. In-Scope Requirements

The test system must support:

• Running all modes through OpenClaw
• Local model execution on DGX Spark
• Frontier model execution through API calls
• Structured JSON output for every run
• Deterministic validation of outputs
• Duplicate lookup against prior items
• Gold-set evaluation against labeled examples
• Cost and latency measurement
• Repeatable runs with state reset between runs
• Logging of prompts, model calls, tool calls, validation results, and final decisions
• Mode 4 custom escalation logic

5. Out-of-Scope Requirements

The first version does not need to support:

• Production RSS feed monitoring
• Automated publishing
• Writing final blog posts or social posts
• Long-term memory or learning across runs
• Community OpenClaw skills
• Multi-user access
• Full MLOps pipeline integration
• Human-in-the-loop UI beyond recording escalation outcomes

6. Test Modes

All modes use OpenClaw as the orchestration substrate.

Mode 0: Frontier Only

OpenClaw runs the workflow using only a frontier model such as Gemini or Opus.

The frontier model receives the task and produces the final structured routing decision.

Purpose: Establish the quality baseline.

Mode 1: Local Only

OpenClaw runs the workflow using only the local model on the DGX Spark.

The local model owns the full task and produces the final structured routing decision.

Purpose: Identify where local-only loop control fails.

Mode 2: Local + Deterministic Validation

OpenClaw runs the workflow using the local model, but deterministic validation is applied to the output.

The local model still makes the decision. Validation can reject malformed or structurally untrustworthy outputs.

Purpose: Determine how much reliability deterministic validation adds before frontier judgment is required.

Mode 3: Frontier Governor / Local Worker

OpenClaw runs a governed multi-model workflow.

The frontier model acts as governor. The local model performs bounded worker tasks such as extraction, summarization, candidate framework mapping, and evidence extraction.

The frontier model makes the final routing decision.

Purpose: Determine whether frontier loop control improves results when local models perform bounded work.

Mode 4: Hybrid Escalation

OpenClaw runs the local model by default. Deterministic validators and escalation rules decide whether frontier governance is required.

The frontier model is invoked only when escalation triggers fire.

Purpose: Determine whether selective escalation can deliver near-frontier confidence at lower cost and latency.

Mode 4 is the target architecture.

7. Required Output Schema

Every mode must produce a structured output matching this schema.

{
  "decision": "ignore | save | escalate | draft",
  "primary_vendor": "",
  "secondary_vendors": [],
  "topic": "",
  "framework_mapping": "4+1 | DAPM | AI Factory Economics | Deterministic Code in the Loop | Buyer Room | Fourth Cloud | none",
  "reason_for_decision": "",
  "evidence": [
    {
      "claim": "",
      "source_excerpt": "",
      "source_location": ""
    }
  ],
  "commercial_relevance": "none | weak | moderate | strong",
  "recommended_action": "none | monitor | social_post | blog_note | client_followup | buyer_room_angle",
  "uncertainty_flags": [],
  "duplicate_check": {
    "is_duplicate": true,
    "related_item_id": "",
    "reason": ""
  },
  "escalation_reason": ""
}

8. Governor Response Schema

When a frontier governor is used in Mode 3 or Mode 4, its response must be structured.

{
  "governor_action": "approve | reject | retrieve_more | revise_decision | correct_mapping | escalate_human | take_over",
  "final_decision": "ignore | save | escalate | draft",
  "corrected_framework_mapping": "4+1 | DAPM | AI Factory Economics | Deterministic Code in the Loop | Buyer Room | Fourth Cloud | none",
  "corrected_commercial_relevance": "none | weak | moderate | strong",
  "corrected_recommended_action": "none | monitor | social_post | blog_note | client_followup | buyer_room_angle",
  "reason": "",
  "additional_retrieval_request": "",
  "human_review_reason": ""
}

When governor_action = approve, final_decision must match the local worker’s decision. Any change to the decision requires revise_decision or take_over.

The governor can overrule the local worker, but it must do so in a form the controller can process and score.

9. OpenClaw Substrate Requirements

OpenClaw must be configured so that it behaves consistently across all modes.

These requirements must be verified before the full build proceeds. The test cannot assume OpenClaw exposes the required controls for state reset, model routing, tool surface control, or trace export. The first implementation milestone is an OpenClaw capability audit.

If OpenClaw cannot support a requirement directly, the build must document whether the requirement can be met through configuration, wrapper code, workspace isolation, or a manual reset procedure. If a requirement cannot be met reliably, the test methodology must be revised before running comparative results.

9.1 State Reset

The system must be able to reset OpenClaw state before each test run.

At minimum, the reset process must clear or restore:

• Workspace memory
• Generated files
• Run artifacts
• Temporary context
• Tool state
• Agent notes
• Any learned or persisted skills used during the run

The goal is to prevent learning, memory, or prior outputs from contaminating repeated runs.

9.2 Model Routing

The system must support explicit model selection per mode.

Required model routing patterns:

• Frontier-only
• Local-only
• Local worker plus frontier governor
• Local default plus frontier escalation

The selected model for each step must be logged.

9.3 Tool Surface Control

The tool surface must be narrow and consistent.

Initial allowed tools:

• Source document reader
• Duplicate store lookup
• JSON/schema validator
• Optional local embedding lookup
• Optional frontier API call

Community skills and broad system-access tools should be disabled for the initial experiment.

9.4 Trace Export

Each run must export enough trace data to support evaluation.

Required trace data:

• Input item ID
• Mode
• Run number
• Model used for each step
• Prompt or instruction template ID
• Tool calls
• Tool results
• Validation results
• Escalation triggers
• Final output
• Latency
• Estimated model cost
• Errors and retries

10. Duplicate Store Requirements

The duplicate check must be grounded in an actual archive, not model intuition.

The duplicate store must include:

• Item ID
• URL
• Canonical URL
• Title
• Vendor
• Publication date
• Content hash
• Summary embedding
• Decision label
• Framework mapping
• Recommended action

Duplicate detection should support:

• Exact URL match
• Canonical URL match
• Title similarity
• Content hash match
• Embedding similarity
• Vendor plus topic plus date-window similarity

The duplicate lookup should return candidate matches. The model may explain the duplicate relationship, but it should not invent the archive.

11. Gold Set Requirements

The evaluation must use a labeled gold set of prior items.

The purpose of the gold set is not to establish objective truth in the abstract. It is to measure whether the system makes routing decisions aligned with the operator’s judgment. For this test, accuracy means alignment with my actual or intended decision pattern.

Target size:

• Minimum: 50 items
• Preferred: 100 items

The gold set must include:

• Items that should be ignored
• Items that should be saved
• Items that should be escalated
• Items that should become draft notes
• False positives
• Real business opportunities
• Duplicate or near-duplicate items

Each item must include labels for:

• Expected decision
• Expected primary vendor
• Expected topic
• Expected framework mapping
• Expected commercial relevance
• Expected recommended action
• Duplicate status
• Related item ID when applicable

The gold set must either reflect the real distribution of the RSS feed or report performance per category. Aggregate accuracy alone is not sufficient because the real feed will likely be dominated by ignore decisions.

11.1 Labeling Process

The initial labels should come from actual prior decisions where possible:

• Items that were ignored
• Items that were saved
• Items that became social posts
• Items that became blog posts
• Items that became client follow-ups
• Items that became business opportunities

For items without a clear historical outcome, I label the item according to the decision I would want the system to make. The label is operator-alignment ground truth, not universal objective truth.

If another reviewer participates, disagreements should be recorded and reconciled. Ambiguous items can be marked as such and used specifically to test escalation behavior.

12. Deterministic Validation Requirements

The validation layer must check:

• JSON validity
• Schema validity
• Allowed enum values
• Required field completion
• Evidence excerpt exists in source text
• Evidence is present for material claims
• Recommended action is consistent with commercial relevance
• Escalation reason is present when decision equals escalate
• Duplicate result includes a related item when is_duplicate = true
• Required fields are not empty
• Step budget has not been exceeded
• Context budget has not been exceeded

The validator should not decide whether the item is strategically interesting. It should determine whether the output is structurally trustworthy.

13. Mode 4 Escalation Requirements

Mode 4 requires a custom escalation gate.

The escalation gate decides whether to keep the run local, retry locally, invoke the frontier governor, or escalate to human review.

Programmatic Escalation Triggers

The system must escalate when any of the following occur:

• Schema failure
• Missing evidence
• Evidence excerpt mismatch
• Invalid commercial relevance / recommended action pairing
• Unresolved duplicate candidate
• Tool failure
• Repeated action
• Step budget exceeded
• Context budget exceeded
• Required field missing

Deterministic Proxy Triggers

The system should implement deterministic proxies for softer failure modes where possible.

The goal is to avoid using the local model to judge its own output. Frontier calls used only to decide whether to call the frontier governor should be minimized, because they weaken the economics of Mode 4.

Initial proxy rules:

Summary instead of decision. Trigger when reason_for_decision has high lexical overlap with the source text but does not include decision-oriented language such as “because,” “therefore,” “recommend,” “should,” “ignore,” “save,” “escalate,” “draft,” “monitor,” or “follow up.”

Weak framework mapping. Trigger when framework_mapping is not none but reason_for_decision does not include any configured key terms associated with that framework. For example, 4+1 mappings should reference a layer, plane, infrastructure boundary, control surface, data plane, execution plane, reasoning plane, or application layer. AI Factory Economics mappings should reference cost, tokens, utilization, throughput, bottlenecks, orchestration, or business value.

Overconfident relevance. Trigger when commercial_relevance = strong but the evidence array does not contain at least one explicit vendor, buyer, client, product, market movement, budget, adoption, or competitive-positioning signal.

Underconfident relevance. Trigger when the source mentions a configured strategic vendor, active client, tracked thesis area, or known business opportunity but the decision is ignore or commercial relevance is none.

High output variance. Trigger when repeated local runs on the same item produce different values for decision, framework_mapping, commercial_relevance, or recommended_action beyond an allowed threshold.

Objective drift. Trigger when the final output lacks a valid routing decision, lacks a recommended action, or primarily answers a different task such as generic summarization, sentiment analysis, or vendor description.

Evidence ambiguity. Trigger when evidence excerpts support more than one plausible decision category, or when candidate excerpts contradict one another.

Commercial ambiguity. Trigger when content relevance and business relevance diverge. Example: the item strongly maps to a framework but has no obvious vendor, buyer, client, or opportunity signal.

These proxy rules are first-pass implementation logic. They will be tuned after the first 10-item test set produces failure examples.

14. Metrics Requirements

14.1 Process Metrics

The system must record:

• Schema pass rate
• Step count
• Tool repetition rate
• Escalation rate
• Correct escalation rate
• Missed escalation rate
• Recovery rate
• Completion discipline
• Latency per accepted decision
• Cost per accepted decision

14.2 Output Metrics

The system must record:

• Decision accuracy
• Vendor accuracy
• Topic accuracy
• Framework accuracy
• Evidence grounding accuracy
• Commercial relevance accuracy
• Recommended action accuracy
• Duplicate detection accuracy
• False positive rate
• False negative rate

14.3 Confidence Metrics

The system must support repeated runs of the same item.

Each item should be run three to five times per mode.

The system must measure:

• Decision variance
• Framework mapping variance
• Commercial relevance variance
• Recommended action variance
• Escalation variance
• Acceptance pass rate

Confidence means repeatable acceptable performance, not subjective satisfaction.

15. Acceptance Criteria

The initial test is successful if it can:

• Run all five modes through OpenClaw
• Reset state between runs
• Produce structured outputs for every mode
• Validate outputs deterministically
• Log traces for every run
• Compare outputs against the gold set
• Measure cost, latency, variance, and accuracy
• Demonstrate where local-only execution fails
• Demonstrate whether deterministic validation improves reliability
• Demonstrate whether frontier governance improves quality
• Demonstrate whether hybrid escalation reduces frontier usage while preserving acceptable accuracy

16. Interpretation Rules

If Mode 1 fails, local-only loop control is not sufficient.

If Mode 2 improves materially over Mode 1, deterministic validation is carrying meaningful control-plane value.

If Mode 3 performs close to Mode 0, frontier loop control is useful even when local models do the bounded work.

If Mode 0 and Mode 3 perform similarly with no cost, latency, or privacy advantage, the local worker may be adding complexity without earning its place.

If Mode 4 performs close to Mode 3 at materially lower cost or frontier usage, the hybrid architecture is validated.

If Mode 4 escalates constantly, the local worker model or escalation rules are not strong enough.

If Mode 4 misses important escalations, the control layer is too weak.

If Mode 2 performs close to Mode 4, deterministic code may matter more than frontier reasoning for this workload.

17. 4+1 AI Infrastructure Mapping

Layer 0: Compute & Fabric / AI Utility Layer. DGX Spark, local memory, local inference runtime, model hosting, KV cache constraints.

Layer 2B: Execution Plane. Local model inference, extraction, summarization, classification, RAG execution, and tool calls.

Layer 2C: Reasoning Plane. OpenClaw orchestration, deterministic validators, escalation gate, frontier governor, authority placement, and loop control.

Layer 3: Application Layer. RSS-to-opportunity triage workflow.

Deterministic code in the loop belongs inside Layer 2C as a structural control mechanism. It validates what can be validated, constrains model behavior, and decides when judgment must be escalated.

18. Initial Build Milestones

Milestone 0: OpenClaw Capability Audit

Before building the full harness, verify what OpenClaw actually supports.

The audit must answer:

• Can OpenClaw state be reset reliably between runs?
• Where does OpenClaw store memory, notes, generated files, and run artifacts?
• Can the model be selected explicitly per run or per step?
• Can local and frontier models be used in the same workflow?
• Can community skills and broad system-access tools be disabled?
• Can tool access be constrained to a narrow allowed list?
• Can prompts, model calls, tool calls, outputs, errors, and latency be exported?
• Can custom validation code be inserted after model output?
• Can custom escalation logic be inserted before frontier calls?
• Can repeated runs be isolated enough to support variance testing?

Deliverable: A capability matrix showing whether each requirement is supported directly, supported through workaround, unsupported, or unknown.

Milestone 1: Minimal Signal Harness

Build the smallest OpenClaw-based harness that can run one input item through Mode 0 and Mode 1 with structured output logging.

This milestone should not wait for the full duplicate store, full trace exporter, or complete gold set.

The goal is early signal:

• Does the local model fail in the expected ways?
• Does the frontier model establish a noticeably stronger routing baseline?
• Are the output schema and prompt instructions workable?
• Does OpenClaw behave consistently enough for the test methodology?

Use 10 manually selected and manually labeled items.

Milestone 2: Schema Validation

Add deterministic JSON schema validation and failure logging.

Milestone 3: Gold Set Loader

Create the gold set format and load the first 10 labeled examples from the minimal signal test.

Milestone 4: Mode 0 and Mode 1 Runs

Run frontier-only and local-only modes against the initial gold set.

Milestone 5: Mode 2 Validation

Add deterministic validation to the local-only path and compare against Mode 1.

Milestone 6: Mode 3 Governor Pattern

Add frontier governor / local worker delegation inside OpenClaw.

Milestone 7: Mode 4 Escalation Gate

Add custom escalation logic and measure frontier usage reduction.

Milestone 8: Full Gold Set Evaluation

Scale from 10 labeled examples to 50–100 examples and produce a comparative results table across all modes.

19. Build Decisions

These are not open questions. They are build decisions that must be resolved at specific milestones.

Must Resolve Before Milestone 1

• Which local model is the first worker model?
• Which frontier model is the first governor model?
• Can OpenClaw reliably reset state between runs?
• Can OpenClaw route explicitly to local and frontier models?
• Can OpenClaw export enough trace data for evaluation?

Must Resolve Before Milestone 4

• How should source items be normalized before entering OpenClaw?
• What is the initial step budget per run?
• What is the initial context budget per run?
• What is the initial gold set distribution?

Must Resolve Before Mode 4

• What threshold defines high output variance?
• What threshold defines a duplicate candidate?
• Should Mode 4 retry locally before escalating to the frontier governor?
• What level of cost reduction is required for Mode 4 to be considered successful?
• What accuracy delta from Mode 3 is acceptable for Mode 4?

This assessment is the readiness companion to Operationalizing AI TRiSM: A CTO Advisor Field Guide. The Field Guide maps AI TRiSM concerns to architecture, authority placement, evidence patterns, and the CTO Advisor frameworks. This document serves a narrower purpose: it helps an enterprise determine whether those ideas have been turned into operating reality.

The Field Guide says what the work is.

This assessment asks whether the work has actually been done.

From Governance Language to Operating Reality

Most enterprise AI governance conversations start in the right place and then stall almost immediately. The language is sound. The intent is responsible. The committees are formed. The policies are drafted. Someone points to AI TRiSM and everyone agrees that trust, risk, and security management matter.

Then the implementation team asks the uncomfortable question.

Where does this actually live?

That is the gap this assessment is designed to expose. AI TRiSM gives enterprises a useful institutional language for AI governance, trust, risk, and security. But language is not an operating model. A control that exists in policy but has no system owner, no runtime enforcement point, no evidence chain, and no escalation path is not a control. It is a hope with a slide around it.

This document is not intended to replace Gartner AI TRiSM. It is intended to help enterprise teams operationalize it. The goal is to determine whether an organization has moved from AI governance intent to AI governance placement.

The Core Question

The core question is not whether the enterprise has AI governance.

The core question is whether the enterprise can prove where AI governance takes effect.

That distinction matters. Many organizations can describe their AI principles. Fewer can show where those principles are enforced in the data plane, operational plane, execution path, model lifecycle, application workflow, and human approval chain. Fewer still can explain what happens when those enforcement points disagree.

That is where AI systems fail in production. They do not fail because someone forgot to write down that fairness, reliability, privacy, and security are important. They fail because responsibility is smeared across architecture layers, vendor platforms, internal teams, and business workflows until no one can explain who had authority at the moment the system acted.

The assessment therefore starts with a simple operating assumption: every meaningful AI control must have a place in the system, a responsible owner, a runtime or procedural enforcement mechanism, and an evidence trail.

If one of those is missing, the control may still be useful. But it is not yet operational.

How to Use This Assessment

This assessment is meant to be used in a working session with architecture, security, data governance, platform engineering, application owners, risk, and business stakeholders in the room. It is not a survey to be filled out by a single governance team. If the answers are too clean, the wrong people are probably answering the questions.

For each domain, the team should answer four questions.

First, where does this responsibility live architecturally?

Second, who owns the decision when there is a conflict?

Third, how is the control enforced before or during execution?

Fourth, what evidence exists after the fact?

The answers should be specific enough that an engineer, auditor, or business owner can trace a real AI-enabled workflow from request to outcome. If the answer is “the platform handles it,” the next question is which platform component, under which policy, with which evidence, and under whose authority.

Readiness Levels

This assessment uses four readiness levels. They are intentionally plain.

Level 0: Aspirational

The organization has governance language, principles, or intent, but there is no consistent mapping to architecture, ownership, enforcement, or evidence.

Level 1: Assigned

The organization has identified responsible teams or platforms, but controls are mostly procedural, manual, or inconsistent across business units and AI workloads.

Level 2: Enforced

Controls are implemented in specific systems or workflows. The organization can show where key decisions are made, who owns them, and how policy is enforced before or during execution.

Level 3: Auditable

The organization can reconstruct material AI decisions, explain authority placement, produce evidence across the lifecycle, and adjust controls as systems, models, agents, and business processes change.

The target is not Level 3 everywhere. That would be expensive, slow, and unnecessary. The target is appropriate readiness for the level of authority the AI system holds.

A low-risk summarization workflow does not need the same evidence chain as an agent that can trigger procurement, alter customer entitlements, recommend credit action, or change production infrastructure.

Domain 1: AI Inventory and System Classification

The first failure mode is not model risk. It is not knowing where AI exists.

Enterprise AI no longer arrives only through approved model development projects. It arrives through SaaS features, copilots, agent builders, embedded automation, data platforms, workflow tools, developer assistants, and business unit experiments. An enterprise cannot govern what it cannot see, and it cannot classify risk if it does not know where AI is acting.

A mature organization maintains an inventory of AI systems, models, agents, copilots, data products, and AI-enabled business workflows. The inventory should not stop at model names. It should include business purpose, data access, authority level, human review points, downstream systems, vendor dependencies, and evidence requirements.

This is where the assessment connects directly back to the Field Guide. The inventory is not complete until each workflow has been classified by authority level. A system that summarizes information does not require the same control posture as a system that routes work, initiates action, influences entitlements, or operates autonomously within policy. The names matter because the enterprise needs one shared vocabulary for deciding when AI is merely informing a human, when it is shaping a material decision, and when it has been allowed to act.

The important question is not “Do we have a list?”

The important question is whether the list changes how systems are governed.

A useful inventory drives classification. Classification drives control. Control drives evidence. Without that chain, the inventory becomes another compliance artifact that ages badly the moment a business unit enables a new feature.

Assessment Questions

1. Do we have a current inventory of AI-enabled systems, including vendor-provided AI features and internal agentic workflows?
2. Does the inventory identify where each system acts in the business process?
3. Has each workflow been classified by authority level using the same classification language as the Field Guide?
4. Does classification change required controls, review paths, evidence, and operational ownership?
5. Can we identify systems where practical authority has exceeded the original approval posture?
6. Who is responsible for discovering shadow AI, embedded SaaS AI, and agentic workflows that did not originate through central IT?

Readiness Signal

An organization is moving from aspirational to operational when AI inventory becomes a control plane input, not a spreadsheet.

Domain 2: Data Plane Governance

Most AI risk starts before inference.

The model may be the visible actor, but the data plane often determines the range of possible outcomes. Retrieval sources, embeddings, metadata, lineage, entitlements, masking, retention, and data movement shape what the AI system can know and what it can expose.

This is why AI governance cannot be isolated inside the model platform. A perfectly governed model connected to poorly governed data is not a governed AI system. It is a well-documented risk multiplier.

In the 4+1 AI Infrastructure Model, the data plane is where meaning and risk live. For TRiSM implementation, that means data governance must be connected directly to trust, risk, and security controls. The organization must know what data the system can reach, why it can reach it, how access is constrained, and how those constraints are preserved when data is transformed into embeddings, summaries, prompts, context windows, or downstream outputs.

Assessment Questions

1. Can we identify the authoritative data sources used by each AI system?
2. Are access controls preserved across retrieval, embeddings, prompts, generated outputs, and downstream workflows?
3. Can we trace data lineage from source system to AI output for material decisions?
4. Are sensitive data policies enforced before context reaches the model or agent?
5. Who owns conflicts between business usefulness and data minimization?

Readiness Signal

An organization is operationally ready when AI systems inherit data governance controls instead of bypassing them through convenience layers.

Domain 3: Operational Plane Placement

The operational plane is where AI intent becomes system behavior.

This includes provisioning, orchestration, scheduling, runtime execution, model serving, RAG pipelines, agent coordination, monitoring, and policy enforcement. It is also where many enterprise AI programs discover that “governance” was never actually connected to execution.

Within the 4+1 model, this domain breaks into three subdomains.

The control plane determines how resources, policies, quotas, and placement decisions are managed. The execution plane determines how models, pipelines, tools, and agents actually run. The reasoning plane determines how judgment, constraint arbitration, model selection, escalation, and cross-system coordination happen.

Most enterprise AI architecture discussions under-specify the reasoning plane. That is where authority drift hides.

A system begins as a summarizer. Then it recommends. Then it ranks. Then it routes work. Then it initiates action. At each step, the AI system accumulates decision authority without a corresponding redesign of controls, evidence, or ownership.

Assessment Questions

1. Where are AI policies enforced: before execution, during execution, after execution, or only through review?
2. Who owns runtime behavior when a model, agent, workflow engine, and business application all participate in the same outcome?
3. Are model selection, tool selection, escalation, and fallback decisions explicit parts of the architecture?
4. Can the organization distinguish execution intelligence from governance authority?
5. What prevents an AI workflow from gaining practical authority beyond its approved design?

Readiness Signal

An organization is ready when it can describe not only what the AI system does, but who or what is allowed to decide that it should do it.

Domain 4: Application and Business Workflow Accountability

AI value shows up in the application layer, but so does accountability.

This is where users experience copilots, agents, recommendations, automated workflows, search tools, summarization tools, and decision-support systems. It is also where the enterprise must decide whether AI is providing information, shaping judgment, or exercising authority.

The application owner cannot outsource accountability to the model team. The business process owner cannot outsource accountability to the platform team. The platform team cannot outsource accountability to the vendor. Each may own part of the system, but the business workflow still needs a clear accountability model.

A useful test is simple.

If this system produces a harmful, wrong, biased, insecure, or materially misleading outcome, who explains it to the customer, regulator, board, employee, or business owner?

If the answer is unclear, the authority model is unclear.

Assessment Questions

1. What business process does the AI system alter?
2. Does the system inform, recommend, rank, route, approve, execute, or enforce?
3. Are human review points meaningful, or are they rubber stamps under operational pressure?
4. Who owns the final business outcome when AI materially influences the decision?
5. Are users told when AI is shaping the outcome and what their recourse is?

Readiness Signal

An organization is ready when application accountability matches practical authority, not just formal ownership.

Domain 5: Evidence Chain and Auditability

Trust is not a feeling. It is the ability to reconstruct what happened.

For low-risk AI use cases, that may mean basic logging and user feedback. For high-authority systems, it means a stronger evidence chain: source data, prompt or instruction context, model or tool used, policy checks applied, human approvals, system actions, exceptions, and post-decision monitoring.

The evidence chain should be designed before production deployment. Retrofitting auditability after a system becomes important is painful and often impossible. By then, the organization may discover that prompts were not retained, retrieval context was transient, tool calls were scattered across systems, human approvals happened outside the workflow, and the final business action cannot be tied back to the AI interaction that shaped it.

Evidence does not need to capture everything. It needs to capture what matters for the system’s authority level.

Assessment Questions

1. Can we reconstruct a material AI-influenced decision after the fact?
2. Do logs capture policy checks, data access, model/tool choices, human approvals, and downstream actions?
3. Are evidence requirements based on authority level and business risk?
4. Is auditability designed into the workflow before deployment?
5. Who decides what evidence is sufficient?

Readiness Signal

An organization is ready when evidence is part of the architecture, not a forensic activity after something goes wrong.

Domain 6: Security and Adversarial Resistance

AI security is not a separate concern from enterprise security. It is enterprise security with new attack surfaces.

Prompt injection, data leakage, model abuse, malicious tool use, insecure plugins, poisoned retrieval sources, agent impersonation, and unauthorized action paths all matter because AI systems increasingly sit between users, data, and operational systems.

The security question is not whether the model can be attacked. It can.

The security question is whether the surrounding architecture assumes the model will be attacked and limits the blast radius accordingly.

This requires a security model that treats prompts, retrieved content, tools, agents, APIs, and outputs as part of the attack surface. It also requires separation between instruction, context, authority, and execution. A model should not be able to grant itself permission just because the prompt or retrieved document told it to.

Assessment Questions

1. Are AI-specific threats included in the enterprise threat model?
2. Can untrusted content influence instructions, tool use, or privileged actions?
3. Are tools, APIs, and downstream systems protected by independent authorization checks?
4. Is there separation between model output and system authority?
5. Are AI incidents integrated into security operations and incident response?

Readiness Signal

An organization is ready when AI systems are designed as hostile-input systems, not trusted conversation partners.

Domain 7: Change Management and Continuous Control

AI governance is not a one-time approval event.

Models change. Prompts change. Data changes. Vendors change. Agents gain tools. Business users discover new workflows. SaaS providers quietly introduce AI features. Regulatory expectations move. The risk posture of an AI system can change without a formal project ever being opened.

That is why TRiSM implementation requires continuous control. The organization needs a way to detect when a system’s behavior, data access, authority, or operational dependency has changed enough to require reassessment.

This is especially important for agentic systems. Adding a new tool to an agent may change its authority more than changing the model. Adding write access may change the risk category. Adding retrieval over sensitive records may change the evidence requirement. Adding autonomous retry logic may change operational blast radius.

Assessment Questions

1. What changes trigger reassessment of an AI system?
2. Are prompt, model, data, tool, workflow, and vendor changes governed differently based on authority level?
3. Can business users modify AI behavior without review?
4. Are production AI systems continuously monitored for drift in behavior, risk, and authority?
5. Who can pause, roll back, or restrict an AI system when risk changes?

Readiness Signal

An organization is ready when governance follows the system after deployment.

The Authority Placement Table

The fastest way to make this assessment practical is to classify AI systems by authority. This table intentionally mirrors the Field Guide classification language. The assessment version is shorter, but the underlying model should remain consistent across both documents.

This table is not a compliance classification by itself. It is a forcing function. It makes the organization say what kind of authority the system actually has.

That is usually where the real conversation begins.

Scoring the Assessment

Each domain should be scored from 0 to 3.

0 means the organization has intent but no consistent placement.

1 means ownership is assigned, but enforcement is inconsistent or mostly procedural.

2 means controls are enforced in systems, workflows, or operating procedures.

3 means the organization can produce evidence, reconstruct decisions, and adapt controls as systems change.

The total score matters less than the pattern.

A high score in policy with a low score in evidence is a warning sign. A high score in model governance with a low score in data plane governance is a warning sign. A high score in security with a low score in authority placement is a warning sign.

The goal is not to win the assessment. The goal is to find the places where governance language has not yet become operating reality.

Buyer Room Use Case

This assessment is also useful for vendors.

Enterprise AI vendors often believe they are selling model performance, developer productivity, infrastructure efficiency, or platform consolidation. Those things matter. But executive buyers are increasingly trying to understand where trust, risk, security, governance, and accountability show up in the actual operating model.

A Buyer Room built around this assessment can expose how CIOs, CTOs, CISOs, data leaders, and platform teams evaluate AI platforms beyond feature lists.

The session can test questions such as:

How do buyers classify AI system authority using the same authority model defined in the Field Guide?

Where do they expect governance controls to live?

What evidence do they need before expanding from pilot to production?

Which risks belong to the vendor, which belong to the customer, and which are shared?

Where does the buyer believe the vendor story is credible, and where does it feel like governance theater?

This maps directly to the Field Guide’s procurement rule: do not ask whether a platform supports AI governance. Ask where the control lives.

That last question is often the most valuable one.

Vendors do not need another generic message test. They need to understand whether their product story maps to how enterprise buyers assign responsibility.

Closing Argument

AI TRiSM is useful because it gives the enterprise a shared language for trust, risk, and security.

But shared language is only the beginning.

The hard work is placement.

Where does the control live? Who owns it? When is it enforced? What evidence proves it worked? What happens when the system changes? Who has authority when the AI system, the human reviewer, the workflow engine, the policy layer, and the business owner all touch the same decision?

That is where enterprise AI governance becomes real.

AI does not fail only because the model is wrong. It fails because the organization cannot explain who had authority when the system acted.

This assessment is designed to make that failure visible before it becomes expensive.