We Rate-Limited Every Customer at Once. nginx Was Doing Exactly What We Asked.

The morning newsletter went out at nine. A few dozen people clicked through to the shop at the same time, which on this site counts as a rush. Within a minute they were getting 429 Too Many Requests after a click or two.

Traffic was modest. The servers were bored. And the rate-limiting blocking was added last week on purpose to stop one aggressive scraper. It had worked beautifully.

It was still working. It just had a different idea of who "one client" was than we did.

A rate limit is only as good as your idea of who the client is

Before touching the limit, ask nginx who it thinks is calling:

tail -n 20 /var/log/nginx/access.log | cut -d' ' -f1 | sort | uniq -c
grep limiting /var/log/nginx/error.log | tail -n 3

tail -n 20 /var/log/nginx/access.log | cut -d' ' -f1 | sort | uniq -c
grep limiting /var/log/nginx/error.log | tail -n 3

Every line in the access log comes from the same address, 127.0.0.5. The error log agrees:

limiting requests, excess: 5.2 by zone "perclient",
client: 127.0.0.5

limiting requests, excess: 5.2 by zone "perclient",
client: 127.0.0.5

Forty customers. As far as nginx can tell, one client.

That address is the edge load balancer. Every hop that terminates a TCP connection replaces the client's address with its own, and every customer reaches this box through the load balancer. The limit zone is keyed on $binary_remote_addr, the TCP peer. So the whole shop shares a single bucket of 5 requests per second.

It worked against the scraper on day one for a reason that nobody noticed: the scraper was the only thing in the bucket at the time. Then the newsletter put forty people in the same bucket, and the limit did its job on all of them.

Where the real client went

The load balancer isn't hiding anything. It appends the client's address to the X-Forwarded-For header on every request it forwards. nginx ignores that header by default, and that is right: it's plain text, and any client on the internet can send one.

The mechanism for using it is the realip module. It replaces $remote_addr with an address taken from the header, but only on connections that arrive from peers you've explicitly listed as trusted. That "only" is the whole story of this incident.

Trust exactly the load balancer

Create /etc/nginx/conf.d/real-ip.conf with three directives:

set_real_ip_from  127.0.0.5;
real_ip_header    X-Forwarded-For;
real_ip_recursive on;

set_real_ip_from  127.0.0.5;
real_ip_header    X-Forwarded-For;
real_ip_recursive on;

The first line is the load balancer's address and nothing wider. Then test and reload:

nginx -t && systemctl reload nginx
tail -n 5 /var/log/nginx/access.log | cut -d' ' -f1

nginx -t && systemctl reload nginx
tail -n 5 /var/log/nginx/access.log | cut -d' ' -f1

The access log now shows customers' real addresses, the limiter keys on them, and the scraper is back alone in its own bucket.

real_ip_recursive on matters once there's more than one proxy in front of you, say a CDN and then the load balancer. The header then carries a chain of addresses. Recursive mode walks it from the right, skipping every trusted hop, and stops at the first address you don't trust. That's the real client.

The fix that's worse than the bug

Here is the line that shows up in half the Stack Overflow answers:

set_real_ip_from 0.0.0.0/0;

set_real_ip_from 0.0.0.0/0;

It also fixes the 429s. Instantly. And it tells nginx to believe X-Forwarded-For from anyone on the internet.

Think about what the scraper does next. It connects directly to the box, bypassing the load balancer, and sets a fresh fake address in the header on every request. It never shares a bucket with itself again. Worse, everything else keyed on the client address is now forgeable too: IP allowlists, geo rules, fail2ban, the audit log you'll need in the next incident.

You fixed a nuisance and shipped a bypass. The checker on our version of this box sends requests straight to the server with forged headers and expects them to be refused, because that's the test a reviewer should have run.

The pattern to keep

Load balancers, CDNs, ingress controllers and service meshes all do this. The moment one sits in front of a service, everything keyed on the peer address starts treating the world as a single client: rate limits, allowlists, geo rules, per-IP logging.

Restore the real address with the platform's mechanism, whether that's nginx realip, the PROXY protocol or the standard Forwarded header. Then restrict who may set it. Trust is per hop, not per header.

And when a protection that "worked last week" starts hurting everyone, ask what changed about the population, not about the rule. The rule was the same. Last week the bucket held one scraper. This week it held the newsletter.

If this were an interview

The follow-ups I'd expect after you fixed it:

1. What is the PROXY protocol, and when would you pick it over X-Forwarded-For?
2. Rate limiting by IP has other problems: carrier-grade NAT, office networks, IPv6 address space. What else could you key on?
3. limit_req with burst and nodelay: what does each one do to a short burst of clicks from a real person?

The third one is the one that separates people who've run this in production. Five requests per second with no burst limit punish a page with five images.

Run it on a real machine. This incident exists as a broken box on TroubleOps: Everyone is rate-limited on a quiet site, root in the browser, about 30 minutes, and a checker that passes only when customers get through and forged headers don't. It's one of 7 in Reverse Proxies and the Web Tier. Ten challenges are free with an account if you'd rather start somewhere gentler.

Have you ever found set_real_ip_from 0.0.0.0/0 in a config you inherited? What it was protecting makes a good comment.