Post cover image

October 23, 2026

The Key Wasn’t the Leak

I rotated an API key, topped up $10 and went to bed. My server handed the new key to strangers before breakfast.

By Faris Hassani

4 min read

I rotated an API key, topped up $10 and went to bed. My server handed the new key to strangers before breakfast.

The second email is the one that gets you.

You can explain the first one away. On October 12 my Anthropic balance went from about $205 to zero in a day. My app's AI search costs roughly a cent an answer, so that's a lot of answers nobody asked for. But it was bonus credit, it was late, and I was tired. I figured the old key had got out somehow and that a new one would sort it.

So: new key. Ten dollars on the account. Paste it into the server config, reload Apache, watch one request come back 200. Around ten at night on the 20th I shut the laptop feeling pretty good about myself.

Two days later, Your API key has been disabled. Anthropic had noticed the key being used somewhere it shouldn't be and turned it off for me. Somebody had already run $7.86 of Opus on it. My app doesn't call Opus. It calls Sonnet and Haiku.

Oct 12   first key, drained to zero         $205.09
Oct 22   new key, Opus 5.5 and Opus 4.8     $7.86
Monthly usage at the end: $213.10 of a $220 spend limit.

Eight dollars is nothing. What bothered me was how fast it went. Only my server had ever seen that key, and two days later someone else was spending it. Whatever I'd fixed on the 20th, it wasn't the problem.

Where it wasn't

Looking for a leaked key at midnight is mostly crossing things off a list. I searched every repo and every branch, history included. I checked my laptop for a stray .env and grepped my shell history. I went through chat transcripts in case I'd pasted it somewhere stupid. I even checked whether my own tooling on the server could see it. It can't; it signs in a different way.

Then the server logs, which should have been the easy bit. Every request in them came from ::1. The sites sit behind a Cloudflare tunnel, so as far as Apache knew, every visitor was localhost. Nine months of access logs, and I couldn't tell my own phone apart from a scanner.

Where it was

The keys were in a file under /etc/httpd/conf.d/, as SetEnv lines. That's how my little PHP proxy picks them up. What I'd forgotten, or never properly thought about, is that conf.d is global. Every site on that server could see those variables, including a boat show website I built for a client and hadn't opened since January.

That site had an info.php in its web root. One line, phpinfo(), which I'd dropped there in January to check a PHP setting and then forgot about. phpinfo() prints the server's environment. Thanks to that global SetEnv, the environment had my OpenAI key, my Anthropic key, my app token and a webhook secret in it, in a nicely formatted table.

Once the logs showed real IPs and I grepped every site instead of only the one I was worried about, the pattern was almost boring. Scanners had been fetching that page for months. One grabbed it on October 11, the day before the first bill. On the 21st, about eight and a half hours after I'd made the new key, a credential scanner went through its usual list:

Oct 21 06:41  GET /.aws/credentials
Oct 21 06:41  GET /.env
Oct 21 06:41  GET /?phpinfo=1

It didn't need the first two. Another small site on the same box had two more phpinfo pages, in case one wasn't enough.

Rotating the key fixed nothing, because the key wasn't the leak. The page was, and every new key I made went straight onto it.

What actually stopped the bleeding

Mostly settings I'd picked months ago without much thought. Auto-reload was off, so when the balance hit zero the attacker hit zero too. If it had been on, October 12 would have kept topping itself up from my card. There was also a monthly spend limit on the account, and it held. And Anthropic's leak detection caught the second key before it got to $8, which I'm grateful for and slightly embarrassed to need.

The money was bonus credit and I'm not asking for it back. This one's on me. Honestly, I'd rather lose $213 learning it this way than find out with a client's key.

The fix, this time

  1. Took down all three phpinfo pages. I moved them out of the web roots instead of deleting them, so I still have the evidence.
  2. Moved the keys out of conf.d into a root-only file (chmod 600) that only the one virtual host that needs it includes. I checked from the boat show site afterwards and the variables are gone.
  3. Made new OpenAI and Anthropic keys after closing the leak this time. The old ones return 401.
  4. Blocked dotfiles like .env and .git on every site.
  5. Turned on mod_remoteip so Apache logs the real visitor IP from Cloudflare's header, trusted only when it comes from the tunnel on localhost.
  6. Changed passwords and rotated the webhook secret, since they'd been on the same page.
# Inside the one site that needs the keys, not in conf.d
<VirtualHost *:80>
    ...
    Include /etc/httpd/private/site-env.conf
</VirtualHost>
# conf.d/remoteip.conf
RemoteIPHeader CF-Connecting-IP
RemoteIPInternalProxy 127.0.0.1 ::1

I also managed one more mistake during the cleanup. I ran a sed command to mask the secrets before printing a config file, and the pattern was a bit too narrow, so it printed two of them in full. I rotated the webhook secret again. If you're going to mask secrets, try the mask on a harmless file first.

What I'd tell myself on October 12

A leaked key is a symptom. If you make a new one before you've found the leak, you're just restocking whoever has it.

Think about who can see a secret, not just where it's stored. A global environment variable is visible to every site on the server, including the one you forgot existed.

phpinfo() is a page that prints your secrets. I knew that in theory. I just never connected it to a file I'd made in thirty seconds nine months ago.

Keep auto-reload off and set a spend limit you could live with losing. And if every line in your access log says localhost, you don't really have access logs.

The fix I thought I'd finished on the 20th took five minutes. The real one took a night and a lot of grep. I'm going to bed.

If you run more than one site on a server, it's worth checking what phpinfo() would say about the others.