The AI Engineering Architecture Deficit

Most "AI integrations" I review today look like 1998 ASP scripting with a chatbot bolted on.

I've been writing production software for 28 years. I've watched three hype cycles collapse under their own technical debt: the "just get it working" web era of the late 90s, the microservices-for-everything cargo cult of the mid-2010s, and now this one, the AI integration boom of 2024 through 2026, where I'm watching senior engineers at companies with market caps well into the billions write code I would have rejected in a code review in 1999.

I want to be precise about the claim, because "AI is overhyped" is a lazy take and not the one I'm making. The models are real. The capability is real. What's broken is that almost nobody is architecting around them. They're wiring them.

What "wiring" looks like

Here's a pattern I've now seen in production codebases at multiple companies this year, with only the variable names changed:

// controllers/ChatController.js — this is not a hypothetical

app.post('/api/chat', async (req, res) => {
  const { message, userId } = req.body;

  const openai = new OpenAI({ apiKey: process.env.OPENAI_KEY });

  const systemPrompt = `You are a helpful assistant for Acme Corp.
    Answer questions about our products. Be friendly. Don't discuss
    competitors. If asked about pricing, say to contact sales.
    Current date is ${new Date().toISOString()}.`;

  const completion = await openai.chat.completions.create({
    model: "gpt-4o",
    messages: [
      { role: "system", content: systemPrompt },
      { role: "user", content: message }
    ]
  });

  const reply = completion.choices[0].message.content;

  // sometimes there's a regex here trying to extract JSON out of
  // a markdown code fence because someone asked the model to
  // "return JSON" and prayed

  res.json({ reply });
});

Read that again as an architect, not as someone excited about AI. The prompt is a string literal baked into a controller. The provider is hardcoded. There's no interface, so swapping providers means a find-and-replace across the codebase. There's no retry policy, no timeout, no circuit breaker, no fallback. There's no test for this route that doesn't hit a live paid API. There's no versioning on the prompt, so nobody knows what prompt was live when a customer complained about a bad answer three weeks ago. And the "structured output" is a regex hoping the model behaved.

This is the same shape as a late-90s ASP page with SQL string-concatenated straight into the request handler. We spent fifteen years teaching an entire industry not to do that: Repository pattern, DTOs, parameterized queries, dependency injection. The AI wave has reset the clock, because the API feels different enough that people forgot the lesson still applies.

An LLM is not a black box. It's an untrusted third-party API with a volatile contract.

This is the mental model shift that fixes almost everything downstream. You already know how to build software against an external dependency you don't control. You've been doing it with payment processors, shipping carriers, and third-party auth providers for two decades. The discipline you'd never skip there is exactly the discipline that's missing here.

• You don't trust the output shape. A charge.succeeded webhook you validate. A model response deserves the same treatment, and most teams skip it.
• You version the contract. You pin an API version with Stripe. Pin, log, and version your prompts the same way, because a prompt is the payload you're sending to a remote system, and remote systems change without asking you.
• You isolate the vendor. You don't call stripe.charges.create() from forty places in your codebase. You wrap it. The same applies to openai.chat.completions.create(), literally, not as a nice-to-have.
• You plan for degraded output, not just downtime. A REST API returning a 200 with wrong data is rare. An LLM returning a fluent, confident, wrong answer inside a 200 is the default failure mode.

Nondeterminism isn't a new category of problem. A deterministic system with a nondeterministic component embedded in it is the same shape as putting a third-party fraud-scoring model or a flaky external service behind an interface with a contract. The fix isn't making the AI part deterministic. It's building deterministic scaffolding tight enough that the nondeterminism stays contained, observable, and swappable. Most AI integrations today have no scaffolding at all. The nondeterminism just leaks straight into the UI.

The cost of skipping this shows up on a predictable schedule. Month one, it ships fast and the demo looks great. Month three, someone wants to A/B test two prompt variants and there's no mechanism to do that without a deploy. Month six, the provider deprecates a model version and half the regex-based JSON extraction breaks in production because the new model formats its fences slightly differently. Month nine, someone wants a fallback provider for cost or latency, and it's a rewrite instead of a config change. I've now watched that exact lifecycle play out at three separate clients this year.

The fix isn't new. It's the same patterns that already survived three hype cycles.

1. Gateway / Adapter — isolate the provider

The single highest-leverage change you can make is also the cheapest: never let a controller, a route handler, or a UI component talk to a provider SDK directly. Every provider call goes through one seam.

// domain/ai/CompletionGateway.ts
export interface CompletionRequest {
  systemPromptId: string;      // reference, not inline text
  variables: Record<string, unknown>;
  responseSchema?: JSONSchema; // contract for structured output
  maxTokens?: number;
}

export interface CompletionResult {
  content: string;
  parsed?: unknown;            // validated against responseSchema
  provider: string;
  model: string;
  usage: { inputTokens: number; outputTokens: number };
  latencyMs: number;
}

export interface CompletionGateway {
  complete(req: CompletionRequest): Promise<CompletionResult>;
}

Now OpenAIGateway, AnthropicGateway, and OllamaGateway all implement the same interface, each reading its prompt from a repository and validating its own output before handing it back. The application layer never knows which one it's talking to. Swapping providers, running an A/B between two, or falling back from a rate-limited primary to a backup becomes a composition change, not a rewrite. This is the Strategy pattern wearing a name that sounds newer than it is.

2. Repository — prompts are data, not code

The prompt in the first example is a template literal sitting in a controller, deployed the same way as the business logic around it, with no history, no owner, and no way to change it without a PR and a deploy. That's backwards. A prompt is closer to a config value or a piece of content than it is to code. It changes on a different cadence, often by a different person than the one who wrote the route handler, and it needs a rollback path when a new version regresses.

// domain/ai/PromptRepository.ts
export interface PromptTemplate {
  id: string;
  version: number;
  render(variables: Record<string, unknown>): ChatMessage[];
  modelOverride?: string;
}

export interface PromptRepository {
  getActiveVersion(id: string): Promise<PromptTemplate>;
  getVersion(id: string, version: number): Promise<PromptTemplate>;
  publish(id: string, template: string, activatedBy: string): Promise<PromptTemplate>;
}

Backed by a table, not a string in git blame. This buys you three things most teams don't realize they need until they're already in a fire drill: a rollback that doesn't require a deploy, an audit trail for what the assistant actually said to a specific customer on a specific date, and the ability to run a prompt experiment without touching application code.

3. Pipe & Filter — the request is a pipeline, not a function call

The most consequential missing piece in most AI integrations is that "call the model" gets treated as the whole operation, when it's really one stage in a pipeline: sanitize the input, retrieve context, execute the call, validate the output against a schema, and only then let it near a user or a downstream system.

// composition — this reads like an assembly line, because it is one
const supportPipeline = new Pipeline([
  new InputSanitizationFilter(),      // strip injection attempts, PII scrubbing
  new RetrievalFilter(vectorStore),   // pull relevant KB chunks, attach as context
  new CompletionFilter(gateway, "support-reply-v3"),
  new SchemaValidationFilter(SupportReplySchema),
  new ModerationFilter(moderationClient), // last line before it reaches a human
]);

const result = await supportPipeline.run(
  { userId, message },
  { requestId, traceId }
);

Every stage is independently unit-testable with no network call. The sanitization filter gets tested with malicious prompt-injection strings and asserts they're neutralized. The schema validator gets tested with malformed model output and asserts it throws a typed error rather than passing garbage downstream. You can insert a caching filter, a cost-tracking filter, or an eval-logging filter without touching the stages around it. This is Pipe and Filter, and it's been running ETL systems and middleware stacks since before most "AI engineers" were born. It doesn't stop working just because one of the stages is nondeterministic.

What this buys you, concretely

Not "cleaner code" as an aesthetic preference. Specific, board-visible outcomes:

• Vendor leverage. When a competitor ships a better model, or a provider has an outage, or a client demands "not OpenAI" for compliance reasons, it's a config change behind the gateway interface, not a migration project.
• Testability without burning API budget. Your CI suite runs against the gateway interface with a fake implementation, in milliseconds, for free. Teams without this either skip testing the AI paths entirely, or burn real API spend on every CI run.
• An actual audit trail. When a regulator, a customer, or your own legal team asks what the system told a user and why, you have prompt version, model, input, and output logged as first-class data, not something reconstructed from a git log and a Slack thread.
• Changes that don't ripple. Prompt tuning, model swaps, and new validation rules land as isolated commits against a stable interface, reviewable by someone who isn't the one engineer who understands the whole tangle.

None of this is exotic. It's ordinary software architecture applied to a component that happens to be nondeterministic instead of a database or a payment processor. It's rare in AI codebases right now not because it's hard, but because the tooling ecosystem shipped fast demos years before it shipped architectural conventions, and most teams copied the demo shape straight into production because nobody senior enough stopped to say this doesn't scale past the pitch deck.

Where this goes wrong at scale

I want to be direct about the failure mode I keep seeing repeated, because it isn't incompetence, it's incentive. A team ships the spaghetti version, it works, the demo lands, the feature ships. Six months in, the prompt has been hand-edited a dozen times directly in production code with no record of what changed or why customer satisfaction dropped after version seven. The one engineer who understood the full request flow, sanitization, retrieval, the model call, the regex parsing, has left. Onboarding a new hire to "how does the AI feature work" takes a week of code archaeology instead of an afternoon reading an interface. And someone is asking why the API bill tripled, and nobody can answer, because there's no cost-tracking filter, no usage logging keyed to feature or customer, just a raw API key making calls from a dozen different code paths.

That isn't a hypothetical roadmap. It's the current state of more than one production system I've been brought in to review this year, and it's entirely avoidable with patterns that predate the web having cookies.

The question worth asking in your next architecture review

If your AI features were built by developers who learned this stack in the last eighteen months, ask one question: if you had to swap your model provider tomorrow, how many files change? If the honest answer is more than one, you don't have an AI feature. You have a liability with a chat interface bolted on top of it, and the bill for that comes due exactly when you can least afford it: mid-scale, mid-fundraise, or mid-incident.

This is exactly the gap Champlin Enterprises exists to close. It's a principal-led engineering firm, every engagement architected and delivered by someone who's been doing this since before "AI engineering" was a job title, not handed off to a junior bench. AI-native systems built the way production software has always needed to be built: gateway-isolated providers, versioned and testable prompts, pipelines instead of prayer-based regex parsing, architecture that survives the next model release instead of requiring a rewrite for it. If your AI integration was built fast and you're wondering what it'll cost to keep alive, that's a conversation worth having before the next incident forces it.