“It worked in staging” fails when WooCommerce checkout isn’t idempotent

The moment it broke: phantom orders during a payment retry storm

Two months ago, on a Fortune 500 apparel brand running WooCommerce behind an edge cache + a modernized WordPress front end, we watched their “success” page spike to normal-looking conversion numbers while the ops dashboard quietly screamed: duplicate order attempts, the kind that don’t always create duplicate final orders… but they do create chaos.

The triggering event was mundane: a transient network issue between their payment processor and their backend webhook endpoint. The processor retried webhooks aggressively (multiple attempts per authorization), and their checkout flow wasn’t built to prove idempotency across retries. Staging had been “green” because staging never had the retry pattern + race timing we saw in production.

In the first 22 minutes, we measured ~0.8% of checkout attempts triggering an “order in progress” path more than once. Not catastrophic—until you notice what those retries do to inventory and cart recalculation. That’s when phantom orders are born: the system can accept the same customer intent multiple times.

My take: staging passes don’t mean correctness; they mean timing didn’t hurt you

Most teams treat staging like a correctness proof. I stopped doing that after the second incident I’ve lived through where staging “passed” because it didn’t reproduce concurrency. WooCommerce makes it easy to ship, but it’s not magic about idempotency. If your checkout can’t prove idempotency under retries, you’re one spike away from phantom orders.

Conventional wisdom says “add a webhook handler and wait until payment is confirmed.” That’s not enough. The real failure mode is this: you need idempotency at the business intent boundary, not just at the payment event boundary.

Failure pattern #1: inventory correctness collapses under concurrent cart mutations

Here’s the specific thing we saw with the apparel brand. The customer added items, went to checkout, and their cart totals were recalculated server-side. Then, during the payment retry window, another request hit a different stage of the checkout pipeline. Depending on which PHP-FPM worker handled it, inventory was decremented once, twice, or not aligned with the final payment confirmation.

Why? Because the WooCommerce order creation + inventory reduction path can be triggered by multiple request paths (AJAX checkout refresh, “place order” endpoint, and webhook-triggered status changes). If you’re relying on “should only happen once,” you’re making a time-based assumption.

Concrete symptom: inventory counters drifted for certain SKUs by up to 1–3 units per 1,000 checkouts during the incident window. That sounds small, but it breaks trust fast: customers see “available” → “unavailable” flips seconds apart.

Fix we now treat as non-negotiable: inventory mutation happens only once, tied to an idempotency key and an order-intent state transition. In practice, we implemented a lightweight “checkout intent lock” table in MySQL keyed by:

• customer id (or anonymous cart token)
• cart hash (items + quantities + selected shipping method)
• idempotency key (generated at checkout start)

Then we ensured that the first successful “place order” attempt transitions intent state from pending → created, and inventory reduction only happens during that transition. Webhooks update order status, but they never re-run inventory logic. Anything else is an implementation bug.

Tradeoff: you add a DB write in the checkout hot path. But we measured it: checkout p95 added ~35–45ms and removed the inventory drift entirely during retries.

Failure pattern #2: race conditions between “place order” and webhook ordering

Webhook ordering is the worst offender because nobody controls it. On that same brand, we saw the “payment succeeded” webhook arrive before the “order created” request fully committed in MySQL. The webhook handler tried to locate the order by a payment reference, updated status, and triggered downstream hooks. Then the “place order” request completed and also attempted to finalize things, leading to double transitions.

Staging doesn’t reproduce this because staging usually has lower load and more stable networking. In production, even with the same code, request completion order changes.

Specific failure: a role-based access edge case amplified the problem. The webhook was processed under a technical user with broader permissions than the checkout user. That meant hooks fired that the checkout request didn’t, because capabilities checked different states. Net effect: some sites “worked” until permission context differed.

Fix: treat webhook handling as eventual confirmation, not as the driver of business transitions. We changed the webhook handler to be strictly idempotent and state-aware:

• Use the payment provider event id as an idempotency key (store event_id in a table with a unique index).
• If the event was already processed, do nothing.
• Never run order creation or inventory reduction inside the webhook; only update order status if the order exists and the status transition is valid.
• If the order doesn’t exist yet, record “event received” and let a background job reconcile once the order appears.

This seems slower on paper, but it’s actually faster operationally. During the incident, the error rate dropped from ~0.8% duplicate attempts to ~0.05% after the changes—because we stopped letting order lifecycle hooks run twice.

Tradeoff: you introduce a reconciliation job. But reconciliation is deterministic; retries are not.

Failure pattern #3: checkout idempotency breaks when cart totals are recalculated mid-flight

Another pattern we hit during modernization work on a regulated beverage portfolio (different client, same class of bug). They had a headless-ish WordPress setup: the catalog and promotions were refreshed via API, but the cart and totals were still WooCommerce-driven. During checkout, totals were recalculated as customers changed shipping address or refreshed the page.

The “place order” request used whatever totals were computed at that moment. Meanwhile, retries would re-run “place order” with a slightly different server-side cart state because cart recalculation re-derives totals (tax, discounts, shipping rules). So even with payment retries, the request was not semantically identical—meaning idempotency keys based on request payloads failed.

Fix: build idempotency around a stable representation of customer intent, not ephemeral totals. We generate an idempotency key at checkout start using:

• cart line items + quantities
• shipping method selection
• discount rule ids (not discount labels)
• tax mode assumptions
• currency

Then we snapshot those values server-side when the intent is created. Subsequent retries validate the cart against the snapshot; if it differs, we return a “checkout changed; re-confirm” response instead of trying to force a payment to match a mutated cart.

That last part matters. Teams often “fix” this by recalculating totals and pretending it’s still the same purchase. It’s not. If you can’t prove that the second attempt is identical to the first, don’t merge them.

Implementation notes: what we do in practice with Woo + Laravel

We’re not dogmatic about “build everything in Laravel.” WooCommerce is their commerce engine; we keep it that way. But we do pull critical correctness logic into a thin Laravel service layer or a dedicated internal module—especially where applied-AI is also in the mix (fraud heuristics, customer support flows, automated incident triage).

1) Storage primitives

• checkout_intents: unique key (customer/cart token + idempotency key), stable snapshot fields, state machine.
• payment_events: unique event_id, provider payload checksum, processed timestamp.

Unique indexes are your best friend. “Check then insert” without constraints is where races live.

2) Enforce idempotency at the boundary

The idempotency check must happen before you create an order or mutate inventory. If you only do it after the fact, you’ve already done the harm.

3) Don’t trust hook order

WooCommerce has its own action/filter landscape. Hook order differences between plugin combinations, caching conditions, or modernized deployments can change timing. In production, I don’t treat hook order as deterministic. I treat it as “best effort” and design idempotency to survive re-entry.

Numbers from our playbook

After we applied these changes across two production sites (one apparel brand, one beverage portfolio) we tracked three metrics for 30 days:

• Duplicate “order in progress” transitions: down from ~0.8% of attempts to ~0.05%.
• Inventory drift events: eliminated (from 1–3 units drift per 1,000 checkouts during retry windows to zero during the same retry pattern).
• Manual support tickets about “charged twice / order status mismatch”: down ~62% during incidents.

The p95 checkout time increased by ~40ms due to intent snapshot writes. That’s a trade I’d make every time because correctness beats speed when payment and inventory are involved.

What to do Monday: a checklist that actually catches the bug

• Simulate webhook retries with randomized ordering (send event B before event A; repeat event N times). If you can’t reproduce your bug locally, you don’t own the bug.
• Attempt duplicate “place order” calls for the same idempotency key and verify: exactly one order, exactly one inventory mutation.
• Snapshot intent and ensure that cart recalculation can’t silently change the purchase after the intent is created.
• Verify permission/context assumptions so webhook execution doesn’t fire privileged code paths checkout doesn’t.
• Use DB unique constraints for idempotency keys; don’t rely on in-memory “already processed” flags.

If your WooCommerce checkout can’t prove idempotency under retries, you’re one spike away from phantom orders.

End-of-day rule: Any “it worked in staging” story that doesn’t include retry storms and webhook reordering is just a story, not an engineering validation.