From 75 minutes to 15: how we ship Atlas now

Getting a finished change from a green pull request to production used to take us more than an hour. On a busy day it was closer to an hour and a half, and almost none of that time was spent doing anything new. This week we rebuilt the release pipeline for the Atlas API, the admin and the layer packages. The first release through it, API v0.1.37, went from the promotion merge to production in 14 minutes and 23 seconds, with nobody touching it in between.

Where the time went

We timed a normal release step by step. The pattern was obvious once it was written down: the same code was tested over and over.

• CI on the feature pull request, about ten minutes. Needed.
• CI again on the push to dev, the same commit.
• CI again on the dev to main promotion, twice, because the push and the pull request runs both fired.
• CI again on the release pull request, which only changes version.txt and the changelog. It also never started on its own: release-please opened it with the default token, and a pull request made that way does not trigger workflows, so someone had to push an empty commit to wake it up.
• A test job inside the image build, and an image build on every push to main, so the promotion and the release each built images.
• Rector, our PHP upgrade checker, timing out on the shared runner with "some parallel jobs have not finished". Every hit cost another ten minute rerun. In one afternoon it hit three times.

Test once

A commit is now tested on its pull request and on the push to dev, and nowhere else. The promotion pull request from dev to main and the release pull request skip the heavy jobs: that exact tree already passed, and the release one only bumps a version. They show a small gate job instead, which passes in seconds, so every pull request still has a green check. A hotfix branch going straight into main still gets the full suite.

Build once

Images are built when a release tag is pushed, not on every push to main. We looked at reusing the image already built on dev, but the API reads its version from version.txt at runtime, and that file changes in the release commit. Building from the tag keeps the version an operator sees true to the code that runs.

The release merges itself

release-please now opens its pull request with a GitHub App token instead of the default one, so CI starts by itself and nobody pushes empty commits. A small workflow listens for that CI run. When it succeeds, it merges the release pull request with a merge commit, but only if the pull request still points at the exact commit that was tested. If anything moved, it does nothing and the next run decides.

The tag deploys itself, with a restore point first

The image build calls the production deploy as a reusable workflow, at the tag's own commit. We chose that over a workflow_run trigger on purpose: workflow_run checks out the default branch, so the deploy would read its stack file from dev instead of the release.

Before any migration, the deploy dumps the production database to a dated file on the host. It then checks the file: at least a kilobyte, and ending with the dump tool's own "Dump completed" line. An empty or truncated dump stops the deploy before a single migration runs. Every existing safety step stayed: health checks, one deploy at a time, and the rollback detection that already caught a failed image pull this week.

A calmer Rector

The Rector timeouts came from too many workers on a runner shared with everything else. We gave it a ten minute timeout, four workers and smaller jobs. It is slightly slower when the runner is idle and finishes reliably when it is busy, which is the trade we wanted.

What a release looks like now

Here is v0.1.37, measured from the moment the promotion was merged:

• 36 seconds: release-please opens the release pull request.
• 52 seconds: its CI passes and the bot merges it.
• About one minute: the tag and the GitHub release exist and the changelog is posted.
• Seven and a half minutes: the images are built and pushed.
• 14 minutes 23 seconds: the database snapshot is taken, migrations ran and every service runs the new image.

The one manual step left is the promotion itself, which is where a person decides a release is ready. Everything after that is mechanical, so it should not need a person.

What we would tell another team

• Time it before you optimise it. We assumed the deploy was slow. It was seven minutes out of seventy five.
• Count how often the same tree is tested. Each extra run felt harmless on its own.
• Make the robot prove it is merging what was tested. Comparing the commit before merging costs one line.
• Never migrate without a restore point you have checked. A dump that exists is not the same as a dump that finished.