Tuning a live API without breaking the sites on it

Before we start the next big piece of Atlas, we wanted the core to be boring: fast, predictable and with no surprises left in it. So we audited the public API end to end and fixed everything we found. The catch was that two brand sites already run on it in production, one of them serving close to two thousand distinct API URLs. Whatever we changed, those sites had to keep receiving exactly the same answers. This is how we did it over six small releases in two days, and what the numbers say now.

Rule one: freeze the answers before touching the code

An audit that changes hot paths needs a referee that does not care about intentions. Ours is a golden snapshot. A small script walks every public URL the live sites use (pages in every locale, entry lists, term archives, menus, settings, the sitemap), about 2,500 URLs in total, and stores each response as canonical JSON next to its status code. It runs gently, a few requests a second, from inside the network so nothing goes through the edge.

Every release got the same treatment: capture, promote, wait for the deploy, capture again, compare. A difference is never waved through. Each one has to be explained by content that really changed. Most releases showed exactly three differences on our own site, all caused by the changelog entry that the release itself publishes. The other site showed zero.

Once the referee earned its keep. After one release, two term archive lists came back with the same entries and the same terms, but in a different order. Nothing in that release touched terms. The cause was older: the query that loads an entry's terms had no explicit order, so the database returned them in whatever order its plan produced, and the plan changed when the cache was rebuilt. No visitor could see it, because no page renders the first term on its own. But an API that answers the same question in two ways is a bug waiting for a client that relies on it. The fix was one line, an order by creation, and the next snapshot came back clean.

What the audit found

• Hot queries without the right index. The page lookup and the scheduler scan ran on indexes that covered only part of their filters. We added six indexes, built online so the live tables never locked.
• One media lookup per image. A page with many images resolved each image on its own. It now loads them in one batch per response, and a fixture compares the batched output with the old output byte for byte.
• The sitemap read columns it never used. It now selects only the fields it writes out.
• Every request looked up the brand and the token in the database. Those lookups are now cached for five minutes. The hard part is revocation: a request that read the old row just before a token was revoked could write it back into the cache right after the revoke cleared it. A short tombstone, written after the revoking transaction commits, closes that window, and a test runs the race on purpose.
• Menus, settings and the theme bypassed the response cache. They now carry the same versioned cache and validators as pages, so a repeat request is answered without a single query.
• Module gates had gaps. A few write paths skipped the check for disabled modules, and module and setting reads repeated within one request. The gaps are closed and the reads are memoised per request.
• Some tables only ever grew. Request logs, delivery logs, AI and autopost run logs, audit rows and failed jobs now have retention windows and a nightly prune that deletes in small batches, so it never holds a long lock.
• Inbound work shared a queue with everything else. It now has its own workers, and our long lived worker tests gained cases for the new state, so nothing leaks from one request into the next.
• Search scanned every brand's revisions. The full text subquery now filters by brand first, so one large brand cannot slow search down for the others.

The numbers

We did not want to claim faster without measuring it. We loaded the latest pre deploy database snapshot into an isolated copy with its own cache, ran the version from before the audit and the current one against it, and timed 150 requests per endpoint inside the application, with the network left out. Read the ratios rather than the absolute numbers:

• A cached page, entry list or sitemap: 2 database queries before, 0 now, about 0.6 ms down to 0.43 ms.
• Menus, settings and theme on a repeat request: 2.2 ms with 5 to 8 queries before, 0.44 ms with none now.
• A list of 100 posts with an empty cache: 37 queries down to 8.
• The sitemap of our largest brand with an empty cache: 71 ms down to 40 ms.

One number went the other way: a first request for a menu or the theme is now 0.2 to 0.6 ms slower, because a miss now also writes the cache. Every request after it pays that back many times over.

We also explained every query a cold page issues against the production schema. The heaviest is the sitemap of the largest brand, a scan of about 1,200 rows that takes 2 ms. That does not need an index today. It will when one brand reaches tens of thousands of entries, and it is written down for that day.

The fastest code still sat behind a slow page

Our own marketing site felt slow while the API underneath it answered in under a millisecond. The site's page cache had disappeared. A routine environment change in the hosting control panel had rewritten the server configuration and dropped our cache rules without a word, so every page was rendered from scratch again: about 300 ms at the origin instead of about 10. We put the rules back, and the deploy now re-adds them whenever they are missing, so the next panel edit cannot quietly undo them.

Then we let the CDN cache HTML too, for a short time, with three exceptions: API routes, social share images, and any request that carries a preview cookie, so a draft can never end up in a shared cache. A publish purges it. Time to first byte from outside went from 300 to 650 ms down to about 100 ms.

How it shipped

Seven issues, seven pull requests, six releases. Each change went through full tests on the integration branch, then a promotion that reaches production in about fifteen minutes. The database is snapshotted before every migration, and the golden comparison runs after every deploy. Small releases kept every diff small enough to explain.

What we would tell anyone doing the same

1. Build the referee before the first fix. A snapshot of real answers beats any amount of reasoning about why a change is safe.
2. Explain every difference, even the ones that look harmless. The one we nearly ignored was a real bug.
3. Measure before and after on the same data, isolated from production, and publish the numbers that got worse too.
4. Every list needs an explicit order on a unique key. Without one, the database is free to change its mind.
5. Check the layers above your code. The slowest part of the stack was a cache rule that someone else's tool deleted.