One Nginx line Showed 212 Customers Each Other’s Accounts. The Code Was Fine.

The first support ticket said: "I see someone else's name on my account page."

Support replied with the usual: clear your cookies, try another browser. Forty minutes and two more tickets later, someone read all three together and noticed they came from three different customers, and that none of them had done anything unusual. They had opened "My account". The page they got belonged to whoever had opened it just before them.

For 71 minutes, the shop's account page was a shared document.

This is a composite postmortem, written for a training incident we run on a real broken box. The company, the people, and the numbers are invented. The failure mode is not. It ships into production somewhere every year, almost always as part of a change described as "just a performance tweak", and almost never gets caught by a test. I'm publishing it in the format we recommend for real postmortems, with a note after each section on why that section earns its place.

Summary

For 71 minutes, customers who opened their account page could be shown the account page of whichever customer had loaded it most recently, including name, postal address and last order. The cause was an nginx response cache, added the week before for performance, configured to ignore the application's Cache-Control: private, no-store and Set-Cookie headers. Cached account pages were served to every customer for up to an hour.

Why it's here: one paragraph a busy executive can read. What happened, for how long, to whom, and why. No blame, and no jargon beyond what's needed.

Impact

• 71 minutes from the first cached page to the purge.
• 1,140 account-page responses served from cache. Matching cache hits to session cookies in the access logs found 212 customers shown another customer's page.
• Exposed: name, postal address, last order number. Not exposed: payment data, passwords.
• Data protection notification filed within the regulatory deadline. The 212 affected customers were contacted individually.

Why it's here: impact is measured, not guessed. "Some users" is not an impact statement. Notice that the logs had to be good enough to support this measurement. That becomes action item 4.

Timeline (UTC)

• Mon 09:40. PERF-77 merged: a response cache in front of the shop, with proxy_ignore_headers Cache-Control Expires Set-Cookie, to raise the hit rate from 4% to 71%.
• Thu 13:52. First support ticket: "I see someone else's name on my account page."
• Thu 14:31. Third ticket. Support escalates. Security incident declared at 14:38.
• Thu 14:46. On-call reproduces with two test accounts. The response carries X-Cache-Status: HIT.
• Thu 14:58. proxy_ignore_headers removed and nginx reloaded. The leak continues: the cached entries are still on disk.
• Thu 15:03. Cache directory purged. The leak stops.

Why it's here: timestamps, not narrative. Look at the gap between 14:58 and 15:03. The first fix was incomplete, and five more customers were exposed while everyone believed it was over. Good timelines keep the mistakes in. That's where the lessons are.

Root cause

The application marks per-user pages Cache-Control: private, no-store and sets cookies on them. A shared cache must never store those responses, and nginx respects both headers by default. PERF-77 told it to ignore them, so every response became cacheable for 60 minutes. The cache key was the URL, so /account was one cache entry for all customers.

Why did the change ship? The hit-rate improvement was measured on anonymous traffic in staging. No test fetched a personalized page as two different users through the proxy. The review checked the config syntax and the benchmark, not the meaning of the overridden headers.

Why it's here: a root cause explains why the system allowed the change, not only what the change did. "Engineer X added a bad line" is never a root cause. Engineer X will be replaced by Engineer Y with the same review process, and Y will add the same line.

What went well

• The X-Cache-Status response header made the cause visible in a single request.
• Access logs carried the session cookie hash, so the impact could be measured exactly rather than estimated.

What went poorly

• Three tickets over 40 minutes before escalation. Support had no "possible data leak" path.
• The first fix didn't purge the cache. Five more minutes of exposure.

Action items

1. Prevent: a CI test that requests every personalised route as two users through the real proxy config and fails on identical bodies. (Platform. 2 weeks.)
2. Prevent: a config lint that rejects proxy_ignore_headers Cache-Control or Set-Cookie in a shared cache without a security review. (Platform. 1 week.)
3. Detect: an alert on X-Cache-Status: HIT for any response that sets a cookie. (Observability. 2 weeks.)
4. Respond: a support macro and escalation path for "I can see someone else's data", paging security directly. (Support and security. 1 week.)
5. Respond: a runbook rule that any cache-rule fix includes a purge step. (Platform. Done.)

Why it's here: every item has an owner and a date, and they're grouped by prevent, detect, and respond. An action item that can't be tracked to done is a wish.

The part people skip

If you only have time to write three sections, write these:

• An impact section precise enough to act on. If you can't say how many customers, you can't say who to notify.
• A root cause that explains why the system let the change through. The line of config is the trigger, not the cause.
• Action items that are owned, dated and split into prevent, detect and respond. Most postmortems have five "prevent" items and nothing under "detect", which is why the next incident also takes 40 minutes to escalate.

And one rule from the timeline worth tattooing on the runbook: removing the bad config does not remove the bad data. Caches, queues, CDNs and replicas all keep what you already gave them. Fix the rule, then purge, then verify with the original reproduction.

If this were an interview

The follow-ups I'd expect after you found the line:

1. Which response headers make a shared cache refuse to store a response, and what does private mean versus no-store?
2. The fix at 14:58 was correct and the leak continued. Why, and what should the runbook say?
3. How would you write a test that catches this before merge, without needing production data?

The second one is the real question. Anyone can find proxy_ignore_headers. The senior answer knows the fix isn't done until the cache is empty and a two-user reproduction comes back clean.

Run the incident yourself. This leak exists as a broken box on TroubleOps: Users see someone else's account page, root in the browser, about 40 minutes, and a checker that fetches personalized pages as two different customers and only passes when they differ. It's one of 7 in Reverse Proxies and the Web Tier. Ten challenges are free with an account if you'd rather start somewhere gentler.

What's the longest gap between "fix applied" and "leak actually stopped" you've seen in a postmortem? That gap makes a good comment.