The races that only appeared at 2:17am—and the tests we added
2:17am after a Friday deploy: doubled invoices
I still remember the alert: a Fortune 500 apparel brand’s ops team called because they saw duplicate invoice records for a handful of WooCommerce orders. Not thousands—just enough to make it embarrassing. The weird part was that the payment processor logs looked clean. The bug wasn’t “charging twice.” It was “recording twice.”
The architecture was straightforward: WooCommerce webhooks landed in a PHP endpoint, then we enqueued a background job to sync invoice data into their internal billing system. During the deploy, we restarted FPM, and—because of how their load balancer retried—the webhook handler ran twice for the same event id.
Conventional wisdom says “use queues + background workers, you’ll be fine.” That’s wrong. Queues only make races repeatable, not safe. If your handler is not idempotent, you’ve just moved the problem from request time to worker time.
What actually broke (the production failure mode)
Two workers tried to write the same invoice row at almost the same moment. We had a unique index, but it was on a field we didn’t populate until after the external call returned. Meanwhile, the “first” worker was still waiting on the billing API response; the “second” worker arrived, made the same external call, and then both tried to insert after the responses came back.
The surprising failure mode wasn’t the insert—it was the cache invalidation that followed. Each worker invalidated the “invoice summary” cache key using the same pattern, but with a TTL jitter implemented in PHP. One invalidation landed slightly later, repopulating the cache with stale data. Customers didn’t see the duplicates in the UI immediately, but ops did, and downstream finance workflows tripped.
Net impact: 0.03% of orders got duplicate invoice rows during that deploy window, and the team lost about 18 hours coordinating manual reversals and reconciliation because nobody trusted the automated linkage.
The fix: idempotency keys before “best effort” retries
We implemented idempotency at the boundary:
- We derived an idempotency key from (webhook event id + store id + target invoice version).
- We stored it in a dedicated table with a unique constraint that’s populated before any external API call.
- We wrapped the “write invoice + update cache” into a single transaction boundary in the job so the cache only refreshes after the invoice linkage is committed.
Now, when the webhook arrives twice, the second job fast-fails without touching external systems. Yes, you still have retries—but you don’t have double side effects.
What tests we added (so it never comes back)
We stopped pretending unit tests would catch this. The bug is a race, and race bugs need race tests.
We added three layers:
- Integration test that simulates webhook duplication by calling the endpoint twice with the same event id and asserting only one invoice linkage row exists.
- Concurrency test that fires two job executions in parallel (same idempotency key) and asserts external call count is exactly one (using a stubbed billing client with a call counter).
- Cache consistency test that verifies the invoice summary cache is populated from committed data, not from an in-flight state.
Implementation detail: for the concurrency test, we used a barrier to synchronize workers—because the only way to catch these is to force the overlap that production timing created. If your test lets the first worker finish before the second starts, you’re not testing a race—you’re testing sequential logic.
The “correct” cache key that still collided
That wasn’t the only time we learned that production timing is a liar. Later on a regulated beverage portfolio (headless WP + Laravel services in between), we had a pricing page that occasionally showed the wrong tier price for a single user session.
We tracked it to a cache key collision: the cache key included a product id and locale, but it assumed the “price calculation mode” was derivable from the tier. It wasn’t. There were two modes—standard and promotional override—and the override came from a separate document store that updated asynchronously. Under load, the promotion update arrived after cache read, so the wrong mode hit.
The conventional move is “add more cache keys.” That’s not always right. Cache keys get long, and you’ll eventually miss a dimension and create a new collision you can’t reason about. The better fix was to make the cache depend on a version token that changes whenever the promotion/override dataset changes.
The number that made it obvious
In one week after rollout, we saw a 1.2% mismatch rate between the displayed tier price and the checkout price for a narrow segment. Every mismatch triggered a support ticket. We didn’t have a “bad algorithm”—we had a “missing invalidation dependency.”
Once we added the dataset version token to the cache key and invalidated on version change (not on arbitrary timers), the mismatch rate went to 0.02%—and those remaining were user sessions that were already open with old JS bundles.
Tests we added for cache correctness
Cache bugs are hard to unit test because the symptoms appear after multiple requests. We added:
- Property-based test for cache key composition: for any combination of (product, locale, override version token), the rendered pricing should match the checkout payload generated server-side.
- End-to-end scenario test with simulated async promotion updates: first request reads price, second request updates promotion, third request asserts the cache now reflects the new version.
This forced the system to treat cache as a function of state, not a function of time.
Applied-AI integration: the kill switch incident
Race conditions aren’t the only production gremlins. On the AI Showcase (Laravel 11 + Livewire 3, Anthropic models), we hit a different failure mode: budget guardrails were correct, but the UX wasn’t.
We had a server-side budget cap: once monthly spend hit the threshold, we rejected new generations. But the Livewire front-end kept showing “generation in progress” because it was waiting on a job completion event that never arrived after the server rejected the request.
Result: users clicked repeatedly, and while we blocked new calls, the UI created enough polling traffic to push our API gateway into rate limiting. In logs, it looked like “the model is slow.” In reality, the model wasn’t being called—we were spending money on noise and confusing users into retrying.
We measured it: during the incident window, we saw ~40% of requests hit the “budget rejected” path, while the model call count dropped to near zero. That mismatch is what finally made the root cause obvious.
The fix: kill-switch must be observable
- We standardized a machine-readable error code for “budget locked.”
- We propagated it through the Livewire event stream immediately so the UI transitions to a non-loading state.
- We added a client-side circuit breaker: once it sees budget locked, it stops polling and shows a single clear message.
The take is simple: if you have a kill switch, it must be part of the same state machine as the UI. A silent rejection is a failure mode.
Tests we added for model pipeline correctness
We created a test harness that mocks the LLM client and asserts:
- No model call is made once budget is exhausted.
- The UI receives the budget-locked state in under 500ms (measured via a mocked event loop).
- Polling stops after the terminal state.
Yes, this is “UI testing,” but it’s really pipeline testing—because for AI systems, the integration is as real as the model output.
What I changed after the second incident
After the invoice double-write and the pricing cache collision, I stopped trusting “implicit safety.” Here’s the rule I use now:
- If a system can be called twice (webhooks, retries, worker restarts), it must be idempotent with explicit keys and side-effect boundaries.
- If caching exists, it must be derived from state (version tokens), not from time assumptions.
- If there’s a kill switch (budget, moderation, feature flags), it must be observable by the client and reflected in the state machine.
That’s the “senior engineer” part: you’re not preventing bugs by adding more code; you’re preventing categories of bugs by forcing contracts between components.
Monday-morning quote: Races, cache collisions, and silent AI rejections don’t fail loudly—we fixed ours by making side effects idempotent, cache depend on state tokens, and kill switches observable in the UI.
We see these patterns across WordPress modernization and our Laravel/SaaS systems: the common thread is making contracts explicit (state, keys, and observability) before scale turns timing into production incidents.