Deploy

How code gets from a merged PR to running on Railway staging, which workflow to use for what, and why there are several.

Everything here reflects the process as of 2026-09-13. For the reasoning behind the design, see Release Management — that is a decision record, not an operational guide, and parts of it describe earlier phases.

The model in one paragraph

Nothing is deployed by merging application code. A release is tagged, the tag triggers a build, the built image's digest is recorded in a manifest (release.json) via a PR, and merging that PR deploys it — each service converged to exactly the digest the manifest pins. The manifest is the single source of truth for what should be running; a deploy refuses to run if what it pins is not what GHCR actually serves, and brings the infra tier up first if it is not already running.

Amber = a human acts (steps 1, 4, 6). Blue = a gate that can stop the release.

You dispatch one workflow (step 1's tag, then Release Manifest and Deploy) and merge one PR. The merge is the approval and the trigger — the deploy reads the committed manifest, main only accepts changes via PR, and nothing auto-merges, so a human still decides exactly what ships. The org ruleset requires 0 approvals, so a second reviewer is convention, not an enforced gate.

Which workflow do I use?

All of these live in fin-infra → Actions.

I want to…	Use	Notes
Release new code (any number of services)	Release Manifest and Deploy	Dispatch it, then merge the PR — the deploy fires on the merge
Cut a version tag for one service	Tag Release	Preview first with confirm unchecked
Cut tags for 2+ services at once	Tag Release (Batch)	Tick ALL for the whole fleet
Restart one service, or a whole tier	Deploy to Staging	The operations tool — blitz-api, backend, infra, all, or a comma-separated list
Rebuild images, scale up, then deploy	Release Staging	The only path that builds and scales up in one run
Bring a scaled-down environment back	Deploy to Staging with all	Deploy and scale-up are the same Railway mutation
Check staging matches the manifest	bun run manifest:drift	Read-only; run locally

Two workflows you should not reach for:

• [DEPRECATED] Deploy from Manifest Diff — superseded by Release Manifest and Deploy, which runs the same script behind the merge trigger. Kept only so existing links resolve, and scheduled for removal. To re-run a deploy, use Deploy to Staging.
• Deploy to Production — scaffolding for the future AWS tier. There is no RAILWAY_PRODUCTION_ENVIRONMENT_ID secret and no production GitHub Environment, so it fails immediately if dispatched. Production does not exist yet.

Releasing new code — the full path

1. Tag

Tag Release (Batch) for two or more services, Tag Release for one.

• bump: patch (no contract change) / minor (new capability) / major (requires action beyond deploy).
• Leave confirm unchecked first — that is a pure preview showing current → candidate version per service, with no pushes.
• The version is computed from the manifest, not from git tag, so it can never race a build that has not finished.

Tags are pushed individually, not in one git push: pushing more than 3 tags at once silently drops every on: push: tags: trigger, with no error.

2. Build (automatic)

Each tag triggers that service's build-push-<service>.yml in blitz / blitz-ui. See Build.

The build gates on version monotonicity before pushing: a tag whose version is not strictly higher than every existing tag for that service fails the build, so no bad-version image reaches GHCR.

Typical durations: backend services ~2–3 min, classicml ~5 min, blitz-ui ~9–10 min (it needs ~6 GB of heap per app and cannot be parallelised on a standard runner). Plan around the UI build — it sets the pace of every full release.

3. Aggregate into a manifest PR

Dispatch Release Manifest and Deploy. The form takes one input, dry_run.

It finds every service whose highest pushed tag is ahead of the manifest, confirms each build finished successfully, reads each digest from GHCR, and opens one manifest PR. Then it emails the outcome.

• Safe to re-run any number of times. If builds are still running it says so and writes nothing — re-run when they finish.
• dry_run=true reports what it would do without opening a PR.
• A single outstanding service is deliberately left alone: the per-build pipeline (update-manifest-from-tag.yml) already opens that PR itself.
• It works for any release shape — 14 services, 2, or a tag pushed by hand. It has no notion of "a batch".

4. Merge the manifest PR

Review it like any other PR. Nothing is automated here by design: a human always merges a manifest change.

Merge one manifest PR at a time

The deploy diffs the manifest against HEAD~1. If two manifest PRs merge back-to-back, the first one becomes invisible and those services silently stay on their old version. Merge, let the deploy finish, then merge the next.

Each merge now triggers its own deploy. Concurrent runs are serialised rather than raced (the workflow's concurrency group), so they queue — but the HEAD~1 window still applies to each, so back-to-back merges still lose the first.

You may also see small single-service PRs from the per-build pipeline. The aggregate stage closes the ones its combined PR supersedes; that is expected, not an error.

5. Deploy — automatic

Nothing to dispatch. Since 2026-09-13, merging the manifest PR deploys it: the workflow also triggers on a merged PR touching railway/environments/staging/release.json.

It re-verifies main's manifest against GHCR, then deploys only the services the manifest diff changed, in dependency order, waiting for each tier to become healthy. The automatic path always waits for health with a 600s per-tier timeout — nobody is there to choose.

Merging is the approval: the deploy reads the committed manifest, main only accepts changes via PR, and nothing auto-merges, so a human still decides exactly what ships. What went away is only the third dispatch.

It fires for any merged PR touching that file, not just aggregate-authored ones — so a hand-edited fix and a rollback deploy the same way. A PR closed without merging deploys nothing.

Re-running a deploy

There is no manual deploy dispatch. mode=deploy was removed on 2026-09-17: its scope came entirely from the HEAD~1 diff, so it was correct only while the release merge was still the tip of main. Once any later commit landed, the diff came back empty and the run went green having deployed nothing — worst precisely when you would reach for it.

Use Deploy to Staging instead. It converges to the full committed manifest rather than a diff, takes a service, tier or comma-separated list, and carries the same verify gate, health wait and staging approval.

6. Confirm

cd railway && bun run manifest:drift

Compares every service's live Railway digest against the manifest. This is the only check that proves Railway is running what the manifest claims — a green deploy run is not the same thing.

The verify gate

Every deploy path ends at railway/scripts/deploy.ts (the automatic path gets there via deploy-from-manifest-diff.ts, which computes the HEAD~1 diff below and then calls deploy.ts <changed-services> --from-manifest), and since 2026-09-10 deploy.tsrefuses to deploy a service whose pinned digest is not the digest GHCR serves.

Why it matters: the manifest's image field is a tag reference, and a tag is a mutable pointer — it can be deleted and recreated, repointed, or its image removed, none of which changes the manifest. The digest is immutable. Comparing them is what turns "the manifest claims X" into "X is really there".

This is not hypothetical. On 2026-09-10 a manifest PR was opened pinning blitz-ui to a digest left behind by a deleted-and-recreated ui-v0.3.0 tag — 47 commits behind the code it claimed to be. Only a digest re-check caught it.

What the gate does not touch:

• Services with no manifest entry — infra (s3, valkey, temporal, temporal-ui) is deployed as-configured, so infra deploys and ordinary restarts are unaffected.
• Entries with a null digest — skipped with a warning; "unset" is a legitimate pre-tag state.

Escape hatches, both explicit: --skip-verify deploys anyway, --no-manifest skips the manifest converge entirely and redeploys whatever image is currently configured. Prefer fixing the manifest.

Deploying honours the manifest by default

Since 2026-09-10 the manifest converge is on by default in deploy.ts. It used to be opt-in via --from-manifest, and the callers disagreed — so the same "deploy" meant either "exactly what the manifest pins" or "whatever image happens to be configured", depending on which workflow you used.

Deploy scope

deploy.ts <target> accepts:

Target	Deploys
all	infra + backend + frontend, in dependency order
app	backend + frontend (no infra)
backend	all 12 backend services
frontend	blitz-ui, blitz-mgmtui
infra	s3, valkey, temporal, temporal-ui
<service>	one service
a,b,c	those services, in their real relative startup order

Deploy and scale-up are the same Railway mutation. A deploy starts a stopped service, so deploying a scaled-down environment brings it back — there is no separate "start".

One subtlety worth knowing: a redeploy re-resolves the image, but a crash-restart does not. A crash-looping service never self-heals onto a fixed image; it needs an explicit deploy.

Local fallback

The same engine, for debugging. Requires RAILWAY_API_TOKEN, RAILWAY_PROJECT_ID and RAILWAY_ENVIRONMENT_ID (all in railway/.env).

cd railway

bun run deploy:all            # infra + backend + frontend
bun run deploy:app            # backend + frontend
bun run deploy:backend
bun run deploy:frontend
bun run deploy:infra

bun run scripts/deploy.ts recon,wfw --dry-run    # preview, no mutations
bun run manifest:drift                            # is Railway running the manifest?
bun run status                                    # per-service deployment state

--dry-run makes no Railway mutations but still resolves every service and runs the verify gate, so it catches a misnamed target or a bad digest exactly as a real run would.

Rollback

Reverting a service's manifest entry in a normal PR is the rollback — to the deploy, that is indistinguishable from any other change. Revert the entry and merge; the deploy fires on the merge like any other manifest change. No tag is created, so the version monotonicity check never fires and cannot block you.

See Rollback for the full procedure.

Known gaps

• HEAD~1 diff window — deploy-from-manifest-diff.ts only diffs the last manifest commit. Largely closed by the merge trigger: each merge now fires its own deploy, so every run's HEAD~1 is that PR's own parent, and the shared railway-staging-mutation concurrency group serialises back-to-back merges rather than letting them race. What remains is the failure case — if one merge's deploy fails and another merges on top, the failed change is in no later diff. The daily Drift Check catches that; bun run manifest:drift is the on-demand version.
• A green deploy can still leave crashed services (fin-infra#74). The waiter stops watching a service the moment it reaches SUCCESS, so one that boots and then crash-loops at app init is never re-observed. Seen twice: agents (2026-08-16), and on 2026-09-13 a run reported one failure while status showed three (ws, wfwpdf, classicml). This matters more now that deploys run unattended — until it is fixed, run manifest:drift (step 6) rather than trusting a green run.
• Container health ≠ serving traffic. The deploy waits for a terminal deployment state, which proves the container started. Backends are private-only and unreachable from a GitHub runner, and blitz-apimgmt / bapiproxy have no /health route, so container-level success is the strongest signal available from outside the network.
• No Prisma migrations run anywhere — not in CI, not at container start. Rolling back an image does not roll back schema. A hard gate for any future production tier.
• release-staging.yml and deploy-production.yml inherit the verify gate via deploy.ts, but their inputs predate the manifest and are shaped around the older "deploy a tier" model.