WooCommerce checkout race conditions: what broke, and what we fixed

Monday 11:42 AM: checkout failed only for some users, only during the lunch spike

I still remember the Slack thread from a Fortune 500 apparel brand: “Payment succeeded, but order status is stuck on pending.” The storefront looked fine. The error logs looked quiet. Then we checked the admin order timeline and saw a pattern: a subset of customers had two order records created within seconds, one got paid, the other never transitioned.

Here’s the brutal part: it wasn’t theoretical. It wasn’t a “corner case” on paper. It was a race condition caused by how our WooCommerce payment callback and order finalization interacted with caching, retries, and concurrency. At peak, checkout attempts were firing in parallel across PHP-FPM workers, and our webhook handler wasn’t idempotent.

We lost real engineering time. For three days, we burned ~26 hours across incident triage, log enrichment, and rollback experiments. Sales impact wasn’t “catastrophic,” but it was absolutely measurable: checkout error rate spiked to 2.7% during the lunch window, with most failures resolving only after manual admin intervention.

The conventional wisdom we rejected: “WooCommerce handles this for you”

Most teams assume WooCommerce’s order lifecycle is robust enough to survive callback retries and concurrency. That’s mostly true—until you add custom payment logic, background jobs, or custom fulfillment steps that run on hooks which can fire more than once.

The real reason this breaks in production is simple: webhooks and callbacks are at-least-once delivery. If your handler isn’t idempotent, the system will eventually create duplicates under load. WooCommerce won’t magically “guess” your payment intent uniqueness for you.

What actually broke (the failure mode we could reproduce)

In our setup, checkout creation and payment confirmation were separated: the client completed payment, then a webhook hit a PHP endpoint to confirm and finalize the order. Under load:

• Two webhook deliveries for the same payment reference arrived close together.
• Both requests passed the “order is still pending” check before either updated the order.
• Both advanced state and triggered fulfillment tasks, and one path left the “real” order record orphaned.
• Because we had object caching enabled and a few transients keyed too loosely, some parts of the admin UI appeared consistent while the underlying order state wasn’t.

The scary symptom was: “payment succeeded” in the payment provider dashboard, but WooCommerce order status didn’t reflect it. That’s the hallmark of a concurrency bug where state transitions happen out of order.

Fix that actually stuck: idempotency keys + a single state transition

We stopped relying on “is pending?” checks and replaced them with explicit idempotency.

1) Compute a stable idempotency key

We used a key derived from the provider payment reference (and currency/amount sanity checks). Example:

Then we stored that key with a strict TTL and “claimed” semantics.

2) Make the webhook handler single-writer

In Laravel (we ran a small sidecar service for webhook normalization, but the critical part is the pattern), we wrapped the “finalize order” in a transaction-like flow:

• Attempt to “claim” the idempotency key using an atomic DB insert.
• If insert fails, we treat it as a duplicate delivery and return 200 quickly.
• If insert succeeds, we load the order, verify amount, and perform one state transition.

Atomic insert matters. You don’t want “check then set” because that recreates the race condition.

3) Remove duplicate-trigger hooks

We also audited WooCommerce hooks and found fulfillment logic attached to a hook that could fire more than once when the order is reloaded/updated. We moved fulfillment to a single “order state changed to paid” path and guarded it with a dedicated “fulfilled_at” meta field so it can’t run twice.

Concrete detection: how we would catch this next time

You can’t detect idempotency failures by watching only error counts. The endpoint returns 200 because the payment confirmation is “valid.” The failure is state divergence.

So we added two kinds of monitoring.

1) State divergence metric

• Count orders in “paid by provider” but not “completed/processing” after 90 seconds.
• Count duplicate order creations per payment reference in a 5 minute window.

Before the fix, we saw duplicate creation spikes aligned with webhook traffic. After idempotency, those duplicate counts dropped to near-zero (we still see occasional duplicates from manual admin actions, but provider-ref duplicates disappeared).

2) Hook execution logging (with correlation IDs)

We injected a correlation ID into the webhook request and logged hook entry points for “order finalized” and “fulfillment started.” That let us answer instantly: was it a duplicate webhook, a double hook fire, or a cache/refresh artifact?

Why caching and transients made the incident worse

We had a couple of transients keyed without enough specificity (some included only order ID, others didn’t include payment reference). Under concurrency, that made the UI look “mostly correct,” which delayed the right diagnosis.

Rule I follow now: if you’re caching anything tied to payment confirmation or order lifecycle, the cache key must include the unique payment identity and the cache must be invalidated on the same transaction boundary as the state transition.

Tradeoffs: what we gave up to get stability

• Idempotency adds a write to the DB per webhook call. We accepted it because the alternative was repeated manual remediation and customer-visible failures.
• We reduced “eager” fulfillment and moved to “paid finalized” only. Fulfillment latency increased by a small amount, but customer outcomes improved.

Net result: the checkout error rate returned to baseline within a day, and the incident stopped recurring in subsequent peak windows.

Preventative checklist (paste this into your runbook)

• Assume every payment callback is at-least-once. Design idempotency from day one.
• Use atomic “claim” semantics (DB unique key / insert) not check-then-set.
• Guard fulfillment with a dedicated state meta field (fulfilled_at / fulfillment_status).
• Audit WooCommerce hooks for multiple triggers on order updates.
• Log correlation IDs through webhook → finalize order → fulfillment.
• Monitor state divergence, not just HTTP errors.
• Cache keys for payment/order lifecycle must include stable payment identity.

Monday-morning quote

If the webhook handler isn’t idempotent, WooCommerce won’t save you—production concurrency will create duplicate state, so we fix it with atomic idempotency keys and single-writer fulfillment.

Champlin

And that’s the same kind of discipline we apply across our client modernization and our applied-AI systems: we design for failure modes, then instrument for detection before the next peak window. Projects