Deploy scope & image digests (decision note)

Superseded — 2026-09-10

The recommendations in this note are now implemented, and its description of current behaviour is out of date. Digest pinning, "deploy only what changed", and a pre-deploy digest verification gate all shipped; nothing deploys from :latest any more.

For how deploys actually work today, read Deploy. This page is kept only as the record of why those decisions were made.

Status: Historical decision note. Superseded by the implementation — see banner.

Context

"Deploy to Staging" (fin-infra/railway/scripts/deploy.ts, via deploy-staging.yml) redeploys services in dependency order. Two facts drive everything below:

1. Deploy == scale-up. Both call Railway's serviceInstanceRedeploy; a redeploy starts a stopped service. So deploying a service scales it up; deploying all brings a scaled-down environment fully back (infra + backend + frontend).
2. Every service is pinned to :latest. A redeploy re-resolves :latest at that moment, pulling whatever GHCR points at then — changed or not.

deploy.ts today does NOT compare image digests. Whatever is in scope is redeployed unconditionally (each re-pulling :latest).

Scope: what each target touches

Target	Redeploys / scales up
all	infra (s3, valkey, temporal, temporal-ui) + all backend + frontend
app	backend + frontend (no infra)
backend	all backend services
frontend	frontend only
infra	s3, valkey, temporal, temporal-ui
<service-name>	just that one service

Only that scope scales up. Deploying blitz-apimgmt alone starts apimgmt only — NOT infra. That is fine when staging is already up, but if staging was scaled down (the nightly auto-down, cron 16:30 UTC / 22:00 IST), a single-service deploy leaves its dependencies (Postgres/RustFS/Temporal) stopped and the service will fail to work. In that case deploy all (or infra first, then the services).

The two questions

Q: Do all image digests change when build-push-all rebuilds all 12?

Yes — effectively every digest changes on every all-build, even for services whose source didn't change. A digest hashes the image's layers, and layers are not reproducible here:

• The shared build/Dockerfile.service does COPY . . over the whole src tree in the prerelease stage — so any repo change alters that layer for every service, which cascades to all downstream layers.
• bun install over the monorepo lockfile is shared; a lockfile change re-hashes the install layer for all services.
• No reproducible-build flags (SOURCE_DATE_EPOCH, pinned mtimes) are set, so bundling output and file timestamps vary run-to-run.

Implication: a digest-compare would find nothing to skip right after an all-build (all 12 differ). It only helps after surgical single-service builds, where the other images are byte-identical to what is running.

Q: Should deploy compare digests and skip unchanged services?

The intent is good (only ~3–4 images change on a typical day; re-pulling 18 wastes bandwidth + time). But:

• Useless after an all-build (all digests changed — see above).
• To implement it, deploy.ts would need, per service: the currently-running deployment's resolved image digest (Railway GraphQL — may not cleanly expose it) AND what :latest resolves to in GHCR now (registry API), then skip if equal. That's two extra API round-trips per service and new failure modes on a script that has already had subtle bugs (the stale-deployment health waiter).

Options

Option	Savings	Complexity / risk
Deploy only changed services (discipline; already supported)	High — deploy 2, not 18	None; zero code
Digest-compare in deploy.ts	Medium — only helps after surgical builds	Real: 2 API calls/service, Railway digest read-back uncertain, new failure paths
Pin image tags to git SHA (not :latest)	Enables true skip-if-same-SHA + reproducible/rollback-friendly deploys	Larger IaC change (everything pins :latest today)

Recommendation

1. Now (zero code): deploy only the services you built. This captures most of the savings — the real waste is deploying all when 2 services changed, not the lack of a digest check. Reserve all for "bring a scaled-down env fully back up."
2. Later (principled): pin :sha tags instead of :latest. This is the better long-term fix for both questions: deterministic per-commit digests, trivial "already running this SHA → skip", and it removes the :latest-re-resolves footgun that let a broken :latest roll out during the 2026-08-14 crash-loop incident. Tie this to how we want rollbacks to work (a SHA tag is a natural rollback target).
3. Digest-compare on :latest is not recommended as a standalone step — it adds moving parts to deploy.ts for savings that only materialize on surgical builds, and SHA-pinning solves the same problem more cleanly.

References

• fin-infra/railway/scripts/deploy.ts, fin-infra/railway/shared/startup-order.ts
• Deploy-waits-for-health + shared startup order: fin-infra #71
• :latest re-resolution footgun (crash-loop incident): blitz #1273 (missing app-code COPY)