Finding the real visitor behind three proxies

Why Atlas Shield first saw every visitor as 10.0.4.8, why the leftmost X-Forwarded-For entry is unsafe, and how to pick the right hop count.

The morning after Atlas Shield went live on our own sites, the access log said almost every visitor came from 10.0.4.8. On the other site it was 127.0.0.1. Shield was blocking the right requests, but it had no idea who sent them. A request budget that counts one address for the whole internet is not a budget, and an address rule that can only block your own load balancer is not a rule. This is how we fixed it, and how you can check your own site.

Why the server sees the wrong address

A brand site rarely talks to the visitor directly. Between the browser and the Nuxt server there are usually two or three hops, and each one opens a new connection. The server only ever sees the last hop. On lifefuture.id the chain is Bunny CDN, then Traefik, then an OpenResty proxy, then Docker's overlay network, which hands the connection over from its own internal balancer at 10.0.4.8. On atlascms.id it is Cloudflare, then LiteSpeed, then Passenger, which talks to Node over loopback, so the peer is 127.0.0.1.

Each proxy records where the request came from in the X-Forwarded-For header, appending the address it received the connection from. By the time the request reaches the app the header looks like client, edge, proxy.

The leftmost entry is a lie you cannot detect

The tempting fix is to read the first address in X-Forwarded-For. Do not. Anyone can send that header themselves, and every proxy keeps what it received and only appends to the right. A bot that sends X-Forwarded-For: 9.9.9.9 would walk through the request budget with a new identity on every request, and could get a real visitor's address banned.

The safe rule is to count from the right. You know how many proxies you run. Each of them appended exactly one entry, so the entry that many places from the right was written by your first proxy, describing who connected to it. That is the visitor, and nothing the visitor sends can move it.

Two settings, one number that matters

In the Atlas layer this is two environment variables:

  • NUXT_ATLAS_TRUST_PROXY=true tells the site to read X-Forwarded-For at all. It stays off by default, because a site that faces the internet directly must never trust the header.
  • NUXT_ATLAS_TRUST_PROXY_HOPS is how many entries from the right to step back.

The hop count is not always the number of boxes on the diagram. On lifefuture.id the OpenResty proxy already resolves the real address from Bunny's ranges and puts it at the end of the header, so one hop is right. On atlascms.id LiteSpeed appends the Cloudflare edge it received the connection from, so the visitor sits two from the right. Our first try there used one hop, and the log showed 172.70.208.101: a Cloudflare edge, not a visitor. Two hops gave the right answer.

How to check your own site

Do not trust the diagram, trust a test. Send a request you know Shield will refuse, with a fake forwarding header, from a machine whose address you know:

curl -H 'X-Forwarded-For: 9.9.9.9' https://your-site.example/wp-login.php

Then open the access log in the Shield dashboard. The newest row should show your own public address. If it shows 9.9.9.9, the hop count is too high and visitors can choose their own identity. If it shows a private address or a CDN edge, the count is too low. Change the number, restart the site and test again. It takes a few minutes, and after that the request budget, automatic bans and address rules count people instead of proxies.

What we changed in Atlas

Rows recorded before the fix still carry internal addresses for up to 30 days, so the dashboard now marks private and loopback addresses as Internal and hides the block button for them. Blocking your own load balancer would block everyone.

//Keep reading

More from the blog

All posts →

//Run every brand from one place.

Start with one brand for free. Bring your team when you are ready.

$ npm install @atlas/client @atlas/nuxt

Free for one brandNo credit cardUpgrade anytime