Why Most AI Architectures Collapse Under Governance

Part 3 of a 4-Part Series on AI in Production

In Part 1, I argued that AI doesn’t fail in the demo. It fails the first time you have to trust it.

In Part 2, I walked through what that looks like in practice. The system works, but you can’t control it. You can’t point to where decisions are made, and you don’t have a clean way to enforce policy before something happens.

The natural reaction is to try to fix that. Add structure. Put a check in the middle. Introduce some form of evaluation before execution.

That sounds straightforward.

It isn’t.

The first thing that breaks is where the logic actually lives.

In most systems, it’s scattered. Some of it sits in prompts, some in application code, some in tool definitions, and some inside the model itself. That works as long as everything is flowing.

But the moment you try to say, “this decision needs to be evaluated before execution,” you realize there isn’t a single place to evaluate it.

You’re not inserting a control point into a system. You’re trying to pull one out of something that was never designed to have one.

You can’t control what you can’t isolate.

Once you try to isolate it, the next problem shows up immediately.

You assume you can evaluate a decision the same way every time.

You can’t.

AI systems don’t behave deterministically. Small changes in phrasing or context can change what tools are used, what data is retrieved, or how an answer is constructed. That’s fine in a demo. In production, it means your evaluation step isn’t checking a known action. It’s dealing with a range of possible actions the system might take.

I ran into this rebuilding the CTO Advisor. The GPT version worked. It was useful. It felt aligned. But I couldn’t tell if it was behaving correctly, because I couldn’t see the path it took to get there.

So I moved to a RAG-based approach—not because it’s better, but because it gave me something to inspect. I assumed that if I could see what was being retrieved, I could start to control what the system reasoned over.

That helped. But it exposed something I hadn’t expected.

Vector search gives you proximity, not structure. It gets you something close, but not necessarily something correct in the context of how the system is supposed to reason. If I wanted consistency, I had to control not just retrieval, but the structure of the knowledge itself.

At that point, it became clear this wasn’t about improving the model. It was about controlling what the model was allowed to reason over.

That leads to the next problem: ownership.

Once you separate proposing, evaluating, and executing, someone has to own each of those steps. Is it the application team? The platform team? The people defining the data? The people selecting the model?

Most organizations don’t have a clean answer. So responsibility collapses back into the application—or worse, into the prompt.

That’s where control quietly turns back into behavior.

And then cost shows up.

In a demo, cost is invisible. In production, it isn’t. Now your system is selecting models, calling tools, chaining workflows, and consuming resources in ways that aren’t always obvious.

At some point, someone asks whether you can guarantee it won’t exceed a budget, explain why it used a more expensive path, or limit what it’s allowed to do in certain contexts.

If your architecture doesn’t have clear control points, you can’t answer that cleanly.

The pattern becomes hard to ignore.

The system works in a demo. It gets integrated into a real workflow. Governance requirements show up. The architecture can’t support them. The project slows down or stalls.

Not because the system isn’t capable. Because it wasn’t designed to operate under constraint.

That’s why adding guardrails doesn’t solve the problem. Guardrails assume there’s a place to enforce them. If decision logic is distributed, execution is tightly coupled, and behavior is non-deterministic, guardrails turn into patches. Prompts get longer. Filters get added after the fact.

None of that creates real control.

So the issue isn’t that AI systems aren’t powerful enough.

It’s that most of them are built for flow, not for control.

Part 4: What a Governed AI Stack Actually Looks Like

You now know what breaks. You’ve seen where the cracks appear and why patching them doesn’t hold.

The next question is harder: what does it look like when you build for control from the start?

That’s where this ends up.