The week we lost to a cache key that wasn’t actually unique
Tuesday, 2:17 AM. PagerDuty fired for AI Showcase—not because Anthropic was down, but because our internal audit log suddenly showed the same answer payload being served to multiple orgs. The weird part: nothing crashed. The UI looked fine. The wrongness hid inside the response bodies.
We traced it to a cache entry that was “unique enough” for our first test environment and not unique in production. Specifically: we keyed a response cache by prompt_hash and model, but we forgot that permissions and guardrails in our system depend on org context (tenant ID + budget bucket + kill-switch state). In a single-tenant dev box, it never mattered. In multi-tenant production, it silently did.
Here’s the cost: we served ~10,400 incorrect responses over 36 hours before the monitoring rule caught the pattern. Most were still “plausible text,” so customer support couldn’t triage by eyeballing. We ended up spending 74 engineering-hours chasing inconsistencies and 19 hours cleaning up cached artifacts (and retrying jobs) across workers.
The failure mode you don’t see until you have tenants
Most teams get this wrong because they treat caching like a performance problem rather than a correctness boundary. If authorization or policy influences the output, then cache keys must include every policy input—directly, deterministically, and versioned.
In our case, the output depended on:
- tenant/org ID (data access rules)
- kill-switch state (whether we should even call tools)
- budget guardrails (how deep the agent is allowed to go)
- system prompt version (we bump this often)
Our original key didn’t include any of that. It was effectively “same prompt + same model = same answer.” That’s true only if policy is constant. Production isn’t constant.
What we did next (and what I won’t do again)
I stopped the bleeding with an emergency invalidation and a hard disable of the response cache for multi-tenant calls. The next step was rebuilding our caching layer with a rule I now treat as non-negotiable:
If the request outcome is policy-dependent, the cache key must be policy-dependent.
Fix pattern #1: “cache key = request signature”
We replaced ad-hoc key composition with a single “request signature” function that hashes a canonical JSON document containing:
- tenant_id
- model + model params
- system_prompt_version
- kill_switch flag
- budget_bucket id
- tool enablement list (even if empty)
- input text hash
That sounds heavy, but it’s cheap compared to an LLM call. More importantly, it makes cache correctness auditable.
// Pseudocode
$signature = [
'tenant_id' => $tenantId,
'model' => $model,
'model_params' => $params,
'system_prompt_version' => $sysVer,
'kill_switch' => $killSwitch,
'budget_bucket' => $budgetBucket,
'tools' => $tools,
'input_hash' => hash('sha256', $input),
];
$key = 'ai_showcase:' . hash('sha256', json_encode($signature));
Fix pattern #2: cache namespaces by version
The second rule: when system prompts or guardrails change, you don’t “update entries”—you change the namespace.
We added policy_v{n} to the prefix so we don’t rely on TTLs to protect correctness. Relying on TTL was the original sin here—because it assumes the key will be right. We couldn’t.
Fix pattern #3: include authorization scope in the signature
One lesson I learned the hard way: even if you think “the same prompt always yields the same answer,” authorization rules can change the visible tool results. Our guardrail layer filters tool outputs based on org entitlements. So the signature includes entitlement scope IDs.
I’m not saying every caching strategy has to hash entitlements forever. I am saying: if you’re caching content that a user is allowed to see, the cache key must reflect the permission model.
Why this cost weeks (not just hours)
Because the bug didn’t manifest as errors. It manifested as valid outputs with wrong provenance. Our initial detection looked like a data quality problem. That’s how you burn days.
For example, our logs showed identical answer bodies for different tenants, but the UI uses streaming and truncation. Support agents didn’t see the overlap until we ran a script that:
- pulled response hashes from both tenants
- grouped by hash
- verified mismatched entitlement IDs
By the time we had proof, we’d already wasted 22 hours debating whether the LLM had returned “the same thing twice.” It didn’t. Our system served the same cached object.
Cross-check: we’ve seen similar “quiet corruption” in WordPress
This isn’t unique to applied-AI. We’ve hit parallel failure modes in WooCommerce and headless WordPress modernization projects—especially when people treat caching as a performance-only concern.
One example: a Fortune 500 apparel brand (platform modernization) had category pages cached at the edge, but the cache varied on a session cookie that didn’t always get forwarded in their CDN config. The result wasn’t a 500 error; it was wrong product ordering and wrong promo badges for certain segments. Error rates stayed near zero.
What fixed it? The same principle, different surface area:
- define cache variation dimensions explicitly (customer segment, currency, promo group, stock visibility)
- encode them into cache keys/headers deterministically
- version them when business rules change
In WordPress terms, this is why I’m cautious with “one-size-fits-all” object caching and fragment caching for WooCommerce. You can do it, but only if you can name all the inputs that affect output.
What we watch now
- Cache hit correctness checks: when a cache hit occurs, we sample whether policy inputs match expected values for that tenant.
- Key cardinality monitoring: sudden drops in effective key uniqueness are an early signal of collision or missing dimensions.
- Repro scripts: a single script replays a request signature and verifies the returned provenance metadata.
- Kill-switch precedence: kill-switch changes must invalidate or bypass caches immediately.
Net effect: after the rebuild, we reduced “silent bad responses” incidents to zero in the following quarter, and we reclaimed ~12 hours/month of manual audit work because the cache is now inspectable.
A clear stance: stop treating caching like an afterthought
People love to say “caching is hard.” I agree. But the deeper issue is that teams treat caching as performance engineering instead of correctness engineering. When policy, permissions, and guardrails matter (they always do), the cache key is part of your authorization system.
Cache keys are not strings; they’re contracts. If you break the contract, your system will keep looking healthy while it quietly lies.
Monday morning, tell your team: If output depends on policy, your cache key must include every policy input—or you’ll ship correctness bugs with 0% error rate.
At Champlin Enterprises, we try to make these “correctness contracts” explicit in the systems we ship—whether that’s WordPress/WooCommerce edge caching during modernization or Laravel-based agent pipelines in our applied-AI portfolio—because the best time to prevent a production weeks-long incident is before the first cache hit happens. See how we handle production-grade systems