The Key Was Right, the Container Was Lying: Three Bugs That Hid Behind a Working System
The key was valid. The account was funded. A direct call to the provider returned a perfectly good completion. And my dashboard sat there insisting the fleet was out of credit.
I had just moved every AI product I own behind a single gateway, 18 of them as I write this so I could switch providers without touching application code. The last step was pasting in a funded API key. I pasted it. The tool told me, confidently and incorrectly, that the account had no credits remaining.
What followed was three separate bugs, each one wearing the costume of a different problem. All three share a family resemblance worth naming, because it is the thing that makes infrastructure bugs so much worse than application bugs: every one of them was a change that appeared to be applied but was not.
Bug one: the container was lying about its own configuration
The gateway runs in a Docker container started with --env-file. My key-rotation tool wrote the new key into that file and then ran docker restart.
Here is the part I knew abstractly and had never been bitten by: --env-file is read when the container is created, not when it starts. A restart reuses the environment baked in at creation. So the file on disk showed the new key, the tool reported success, and the process inside the container went on using the old exhausted one. Every check I could think of agreed with itself. The file was right. The service was healthy. The answer was wrong.
The diagnostic that broke it open was comparing the file against the running process rather than against my own expectations:
docker inspect ce-ai-gateway --format '{{range .Config.Env}}{{println .}}{{end}}' | grep OPENAIThe container held a key that ended in different characters than the one in the file. Everything downstream of that was explained in a second.
The fix is not "remember to recreate the container." The fix is that the tool now recreates it, then verifies the running container holds the new value, and refuses to report success if it does not. A tool that can lie to you once will lie to you again at a worse time.
Bug two: a credential contract that changed under me, hours later
The console for the gateway has its own login. Early on I set a username and password for it, applied them with that same ineffective restart, and then built a single-sign-on bridge that logged in using the fallback credential the service accepts when no username is configured. It worked. I tested it. I moved on.
Hours later, fixing bug one, I properly recreated the container. That finally applied the username and password I had set. The service switched to requiring exactly those, and started rejecting the fallback credential my bridge used. Sign-in broke, several hours and many commits after the change that actually caused it.
This is the nastiest shape a bug can take: a latent configuration change with a delayed fuse. The cause and the symptom were separated by so much time and so many unrelated edits that the obvious suspects were all innocent.
The durable fix was not to swap in the new credential. It was to make the bridge try both, so that whichever mode the service is running in, sign-in works. When you cannot be sure which contract is in force, handle both and stop guessing.
Bug three: correct data, wrong on the screen
With the fleet finally live, the dashboard showed total spend of $0.00. Except it had spent money. Sub-cent amounts, but real ones. I was formatting currency at two decimal places, so every genuine number collapsed into a convincing zero.
I fixed the precision and introduced a better bug. The label now read (0.33¢ so far). That is a third of a cent. Kevin read it as thirty-three cents, which is exactly how any normal person reads a decimal next to a currency symbol, and asked whether the pricing was broken. It was not. The presentation was.
Two lessons fell out of that. First, a rounding rule is a truth-telling decision: if your display can render a live number as zero, it will, at the exact moment you most need to know the number is not zero. Second, never restate money in a unit your reader does not think in. The panel now shows dollars with enough precision to move, and says "less than one cent" in words. It also matches the underlying console's precision digit for digit, so the two can never look like they disagree.
The pattern underneath all three
Application bugs usually announce themselves. Something throws, something 500s, a test goes red. These three did the opposite. Each one produced a system that looked healthy from every angle I was checking:
- A config file that was correct, in front of a process using something else.
- A credential that was correct, for a contract that had silently changed.
- A number that was correct, rendered in a way that read as false.
The common failure is checking the artifact instead of the behavior. I checked the file, not the process. I checked that login worked once, not that it would keep working after a rebuild. I checked that the number was right, not that it was readable.
The rule I took away and wired into the tooling: verify the thing that actually serves the request. Not the file that configures it. Not the last time it worked. The live path, right now, end to end.
What that looks like in the tooling now
The key rotation tool validates a new key with the provider before touching anything, applies it by recreating the service, confirms the running process holds it, and then makes a real one-token call to prove billing works. That last step matters more than it sounds: "the key is valid" and "the account can actually pay" are different questions, and only the second one determines whether your product works.
When I finished, I did not declare victory from the gateway logs. I ran all 18 products through their own code paths and made it answer. One reported a failure, which turned out to be my test calling a feature name that product does not define. That is the good kind of failure: the system correctly refused something that did not exist.
A shorter way to say all of this
Infrastructure lies more politely than application code does. It does not throw. It hands you a config file that agrees with you, a health check that passes, and a number that is technically accurate, while doing something else entirely underneath.
Build the check that closes that gap, and build it into the tool rather than your memory. You will not remember the container semantics at midnight. The script will.