Checkout bugs that hide in cache and permissions are the real threat

Monday 2:13 AM: the checkout kept “working”… and still cost us sales

We hit a failure mode that’s brutally hard to spot in staging: a WooCommerce checkout path where “everything looks green” but a small subset of users intermittently fails to complete payment. Not because Stripe was down. Not because the cart was broken. Because two requests raced on the same server-side state and cache made it worse.

This was a Fortune 500 apparel brand (think large catalogs, heavy promotions, and peak traffic patterns you only see on weekdays). We were modernizing their WordPress + WooCommerce stack and tightening performance. The decision that mattered: we enabled aggressive caching around non-checkout pages (full-page caching at the edge), and we also introduced a new “request coalescing” micro-optimization for product/price reads inside WordPress.

At peak, we saw an increase from baseline completion success of ~99.2% to ~98.7%. That’s only a 0.5% hit, but on a high-volume sales day it meant hundreds of lost checkouts. Even worse, the issue wasn’t consistent—customers complained “the payment went through but the order never showed,” and logs were a mess because the failure happened after the payment provider call, during order persistence and stock updates.

The surprising failure wasn’t a classic “broken payment.” It was an application-level race condition between order creation and stock decrement happening under concurrency when cache invalidation and request coalescing weren’t coordinated with the transactional bits.

What broke in WooCommerce (and why conventional advice didn’t help)

Most teams treat cache as “read-only problem” and permissions as “admin-only problem.” That’s the conventional wisdom—and it’s wrong for checkout.

Here’s the chain we saw:

Checkout page loads quickly because edge caching is aggressive for everything except the final submit endpoint.
We added a performance layer that reduced repeated price/meta reads during the session lifecycle (the “request coalescing” part).
During the “place order” request, WooCommerce writes order rows and then updates stock (depending on configuration).
Under concurrency, our coalescing layer leaked a cached view of product stock meta into a code path that should have been “fresh at write time.” That meant stock was sometimes decremented twice or decremented against stale quantities.
WooCommerce then tried to recover, but recovery depended on transient state that had been invalidated inconsistently.

In other words: a caching read contaminated the transactional write path. The app remained functional, but behavior drifted at exactly the moment reliability matters.

Guardrail #1: treat checkout as a “transaction boundary” and ban cache reads there

Our fix wasn’t “turn off caching everywhere.” That’s the lazy answer and it costs money in performance. We changed the architecture of the cache discipline:

We introduced an explicit “checkout boundary” in our code: any request that enters the order-creation flow cannot read from any request coalescing/cache layer that isn’t explicitly designed for write-after-read consistency.
We used database transactions (where possible) around the order creation + stock update logic, and we avoided custom caching wrappers on those code paths.
We added idempotency protection for the “place order” payload: if the provider callback retries, we detect duplicates by payment intent + customer + cart signature and short-circuit without double writes.

Concrete outcome: after the guardrails shipped, completion success recovered to ~99.6% (back above baseline) and the checkout-related failure rate dropped from ~0.5% to <0.1% on the same traffic profile.

Headless ISR cache invalidation: the page refreshed, but the data didn’t

Fast forward: separate engagement, same lesson—cache invalidation bugs only show up when the system has multiple sources of truth.

In headless WordPress setups, we’ve used ISR-style regeneration for marketing pages and product landing content. On paper it’s clean: a request comes in, the CDN serves a cached static page, and the page regenerates in the background for future requests.

In practice, we shipped a stale-cache bug that caused “updated content not matching checkout pricing rules” for a tiny window. A regulated beverage portfolio had promotion rules that changed pricing logic and also changed eligibility text shown on landing pages. During one promo cutover, the landing page that said “Eligible now” remained cached while the cart rules had already changed.

The timeline was tight: content regeneration ran, but the underlying CMS data fetch cached in a separate layer didn’t invalidate because the cache key didn’t include all inputs (eligibility rules were stored in a different ACF field group and mapped to a derived “eligibility summary” that wasn’t part of the invalidation signature).

We saw it for about 37 minutes after the cutover window—short enough to be dismissed as “confusing customer support,” long enough to create refund tickets.

What broke (the failure wasn’t ISR—it was “invalidation completeness”)

Here’s the part that teams get wrong: ISR gives you a refresh schedule, but it doesn’t guarantee semantic correctness unless your invalidation model matches your data dependency graph.

The real issue was that we had multiple caches:

CDN page cache (ISR)
API response cache for headless WordPress data
Derived eligibility summary cache

We invalidated page cache on post update, but eligibility summary depended on ACF field sets that were modified via a workflow action (not always tied to the same post update event). So regeneration happened with stale derived data.

Guardrail #2: make cache keys reflect dependency inputs, not just endpoints

We implemented a dependency-aware invalidation strategy:

Cache keys include a “content fingerprint” that covers the ACF field values actually used to render the page (not just the post ID).
We added a structured mapping from ACF field groups to the API response objects that depend on them.
On workflow actions, we explicitly bump a version number for derived data used in eligibility displays.

After this change, promo cutovers aligned within seconds, and we reduced “landing page mismatch” support tickets by ~62% during the following promo window.

ACF edge cases: the bug lived in the template fallback, not the field

ACF problems rarely look like “ACF is broken.” They look like something is missing—until you realize the field type’s fallback behavior differs between admin preview and published output.

On an agency-managed site, we updated a component that pulled ACF repeater rows and rendered them into a product comparison table. In admin preview, it looked fine. On the public site, one row vanished intermittently. Not always. Only for specific field configurations and only after an editor saved changes quickly.

The failure: a race between autosave revisions and field hydration. ACF saves revisions in a way that can momentarily present partially saved repeater structure to frontend renderers, especially when you combine it with caching and “render on request” headless fetches.

Most teams fix this by “adding sleeps” or by disabling autosave—which is the wrong lever.

Guardrail #3: never trust partial repeater state; validate shape before render

Our rule now: when rendering ACF repeaters (or any complex field shape) from PHP or an API, we validate the shape and fail closed.

If the repeater array is missing expected keys or row counts don’t match the last known good fingerprint, we render a safe fallback and log an event.
For headless APIs, we require a minimum “hydration completeness” check before returning the payload (so the client never receives half-formed content).
We pinned template logic to explicit field schemas rather than “best effort” defaults.

That single change removed an entire class of “invisible content” issues. It also sped up debugging because the logs told us whether the field payload was malformed versus the template being wrong.

Multisite role-permission surprises: the ghost admin is real

Multisite is where permissions bugs become embarrassing, not because they’re hard, but because they’re non-obvious. You can be an admin on the network, and still lack capability on a site—then something “almost works” until a specific workflow triggers it.

We saw this in a WordPress multisite deployment for internal tooling (adjacent to a regulated process workflow). The symptom: users could view pages, but when they clicked “generate report,” permissions failed only for one role. The error appeared as an intermittent 403, and the logs looked like caching again—until we traced capability checks inside a custom plugin.

Root cause: a role capability was granted at the wrong layer. Network-level role assignments didn’t match the site-level capabilities that the code path checked (a classic mismatch between capability checks and the actual capabilities stored for a given blog ID).

Then add a second ingredient: object caching. A cached capability result outlived role assignment updates, so the “ghost admin” persisted for a while.

Guardrail #4: permission checks must be blog-aware and cache-aware

We changed two things:

All capability checks include the correct blog context (current_blog_id) and never rely on global role state.
Any permission-affecting operation bumps a cache version keyed by blog ID so that object caching can’t serve stale capability data.

After this guardrail, the 403 rate in that workflow dropped from 0.9% to 0.02% during the next rollout, and the “intermittent” complaints stopped.

Where applied-AI integration fits (and where it shouldn’t)

We run applied-AI systems alongside these web stacks—Vantage AI, Diamond AI, and the AI Showcase—so we’re used to mixing AI services with production workflows. The temptation is to let AI “help” with checkout or content generation logic. I don’t do that.

The reliability rule is simple: AI systems can read and propose, but they can’t be in the critical path for order placement, stock changes, permission enforcement, or cache invalidation correctness.

Our guardrails for AI jobs mirror the web reliability guardrails:

Hard kill-switch + budget guardrails (no runaway costs when a model misbehaves).
Outbox pattern for side effects (generate content or insights asynchronously; never gate checkout).
Deterministic fallbacks when upstream caches are stale or when field hydration fails.

This isn’t theoretical. The moment you allow AI output to decide transactional state, you create a new class of “it only fails when it’s fast” bugs—because model latency and retries interact with caches and concurrency.

The reliability playbook we ship by default

We now treat checkout, ISR regeneration, ACF hydration, and multisite permissions as four edges of the same reliability problem: hidden coupling between read models (cache, previews, inferred fields) and write models (orders, roles, derived payloads).

Our default guardrails:

Checkout boundary: ban cache reads from transactional write paths.
Dependency-aware invalidation: cache keys reflect semantic inputs, not just endpoints.
Field shape validation: render fails closed when ACF payloads are partial or malformed.
Blog-aware permissions: capability checks and cache versions must include blog context.
Observability over guesswork: log fingerprints for payload completeness and cache dependency versions so debugging isn’t archeology.

We didn’t get reliability by adding more automation or “more caching.” We got it by respecting boundaries: transactions, dependencies, schemas, and scopes.

Monday morning quote: “When bugs only show up in checkout and cache and permissions, it’s not three problems—it’s one missing boundary between read models and write models.”