Deterministic AI Is an Architecture Problem, Not a Model Problem

Ask Claude Code to stand up a Virtual Private Cloud (VPC) and it will. Ask again tomorrow and it will again, but the result will look nothing like the first one: same request, same outcome on paper, completely different artifact underneath. That gap is the whole problem with moving agents from demo to production. The model is probabilistic, so it hits the target you asked for, but not the same way twice. In infrastructure, “not the same way twice” is just a longer way of saying “outage waiting to happen.”

I got into this on the podcast recently with Adam Jacob. He built Chef. He spent a decade teaching enterprises how to think about repeatable automation, so he has watched this movie before. His new company, Swamp Club, is a bet on a specific answer to the problem. The answer is worth stealing whether or not you ever touch his product. The short version: you don’t make the model deterministic. You wrap it. What follows is how I’d put that into practice, backed by what I’m seeing in the lab.

Stop trying to make the model behave

The instinct most teams have is to tighten the prompt. Write a better SKILL.md, add more context, spell out every rule. It helps, right up until it doesn’t. As Adam put it, every time an agent opens a fresh context window it “has to rediscover the whole of the world from first principles.” Did you mean a new VPC or the one you already have? Do you have Amazon Web Services (AWS) credentials? It re-derives all of that, every run. The cost of nudging that probabilistic trajectory toward your intent is ever more token consumption inside ever more loops.

That spiral has a price tag. Adam pointed to David Cramer at Sentry, who spent north of $10,000 on tokens in a week. One code-review skill Adam tested consumed about 4.6 million tokens on a single problem. Making part of that process deterministic dropped it to around half a million, and it ran twice as fast. He wrote the whole thing up, numbers and all. Think about what that buys you. Nothing. You’re paying frontier-model rates to re-solve the same problem from scratch, over and over, and you still can’t reproduce the result.

So here’s the reframe. Don’t spend the intelligence re-solving your problem on every run. Spend it once, to build the machine that solves your problem. Then run the machine. Adam calls it “the machine that builds the machine.” Once you see it, you see it everywhere.

Put the judgment in a box the code controls

Here’s the pattern, concretely. Take any repeatable process. Mine is scoring a vendor against my 4+1 AI infrastructure framework. Most of it isn’t probabilistic at all. Step one, do this; step two, do that; if this, then not that. Computers have been excellent at that kind of flow since forever. So encode it as code, not as a paragraph of hopeful instructions.

Inside that deterministic flow there are a few moments that genuinely need judgment. Fetch the architecture data for this layer, read it, and decide whether it’s good or bad against my criteria. You can’t write a plain program for that. It’s the expert-system trap, and we already learned that encoding every branch of human judgment by hand doesn’t scale. So at exactly those moments, and only those, you call the Large Language Model (LLM). Here’s the load-bearing move. You make the output of that call a known, typed piece of data. Something you can validate. Then you chain those validated steps into larger behavior.

Do that and the agent becomes deterministic in the way that matters. The path it runs on is a program. The creativity lives in bounded, checkable steps along that path. Run the workflow again with new inputs and it executes the same way every time, because that part is just software working the way software has always worked. Now the strange part. You don’t hand-write that machine: the agent writes it, using a fixed architecture you hand it. You’re using AI to make AI deterministic. Sit with that for a second.

The test decides done, not the model

This is the piece my lab work keeps confirming. Once the LLM’s output is data, you can validate it. And validation, not model confidence, is what decides whether a step is finished. In Lab 003 I built a three-tier escalation chain on an NVIDIA DGX Spark: a local model, a mid-tier, and a frontier model, gated by a deterministic test harness, every call metered. The naive read was that escalation to the frontier tier was rescuing correctness. It wasn’t. Several of the harness checks were invalid, and they’d booked “escalations” that were really unsolved cases sitting on broken tests. Fix the tests and the picture flips. The local model was clearing the solvable fixes on its own, and the frontier tier bought throughput, not correct answers. The test still decides done.

That has a direct budget implication. The common pitch is that local agents need frontier-tier escalation to be correct. It didn’t hold. Correctness came from the validator. Reach for the bigger model for speed and volume, not to buy your way to right answers.

It also changes what refactoring costs. Agents are cheap enough to write comprehensive black-box acceptance suites you would never have staffed with humans. Remember the rooms full of Quality Assurance (QA) engineers working a binder, page by page? Gone. Build the test suite first. Then change the architecture underneath and loop until it goes green again. A dramatic refactor stops being high-risk. In Lab 004, where I tested whether frontier models could author production migration patterns, the failures clustered in one place: where the model became the author of correctness. Encoded patterns and human-defined validators are what made migration safe. Not a smarter model.

What this means for the humans

The job isn’t reviewing the code anymore. Frontier models write good code. Babysitting every diff is a poor use of a senior engineer. The scarce skill is the thing I just described. Decomposing messy business requirements into systems that are testable and deployable, deciding where judgment is allowed and how its output gets checked. That’s architecture. It doesn’t disappear because the code writes itself. If anything, it becomes the whole game.

It also inverts how people learn the craft. We used to reach architecture from the bottom up. Years as a plumber before you could see the whole building. Now you start from the architecture and keep the system supple as you learn where you got it wrong. The signal that your design is off no longer comes from reading source. It comes from the top down. A cluster of bugs in one area, a change that’s always painful to make, an outcome the system can’t produce cleanly: those are the tells that the internal architecture is wrong, and you go investigate from the outcome inward.

I didn’t need to know how to write a Transmission Control Protocol/Internet Protocol (TCP/IP) stack to build software that works across a network. There are protocol engineers who live in that detail, and it’s enormously valuable. But we have far more network architects than protocol engineers, because the layers let most of us work above the wire. Deterministic AI is going to draw a similar line. My bet: the architecture role explodes as more teams realize they can make AI reliable and actually capture the productivity, instead of marveling at expensive magic.

Where I’m taking this

I’m running this pattern against real questions in the lab right now. How to test the deterministic loop. How to make migrating from one platform to another possible with AI in a way you can actually reproduce. If you want the lab notes as they land, they’re at labs.layer2c.com. And if you want the full conversation with Adam, it’s the longest episode I’ve run in a while, for a reason. This is the real problem sitting under every stalled AI project. The teams that get past it won’t be the ones with the cleverest prompt. They’ll be the ones who built the machine that builds the machine, and wrote the test that decides when it’s done.