Release Management (design proposal)

This is a decision record, not an operational guide

It documents why the release process is shaped as it is, phase by phase, and parts of it describe earlier states that have since shipped. For what to actually click today — which workflow does what, and in what order — see Deploy.

Status: Decision note — design settled, all seven original open decisions made. Phase 1 is live (tag-triggered builds, health routes); Phase 2 minimal scope is merged (release manifest type, seed file, opt-in Railway image converge); Phase 2.5 is merged (the manifest is now the sole source of truth for image version — infrastructure.ts declares infrastructure only); retroactive-branch patching (rulesets, backport action, ancestry check) is merged; Phase 4 (AWS preprod parity) is merged; GHCR retention (item 7) is resolved; tag-time manifest automation + manifest-diff-driven deploy (v0.19) is merged and live-verified for the entire fleet (v0.20, 2026-09-06) — every one of the 14 Railway staging services has now gone through the full tag → build → GHCR verify → manifest PR → deploy cycle at least once, independently confirmed via Deployment.meta.imageDigest, not just workflow checkmarks. See Phased rollout for exact PR references and what's still ahead — remaining work is AWS (Phases 5–6) and the explicitly-deferred Phase 3 (health-gated auto-rollback). Scope: Railway (dev) + AWS ECS preprod + AWS ECS production. No EKS; Lambda considered and ruled out (all deployable units are long-running HTTP servers or Temporal workers, not short-lived request/response functions).

Why this exists

Today every service on every environment is pinned to :latest. A redeploy re-resolves whatever GHCR points at at that moment — there is no way to say "this environment is running version X of every service" and no way to revert one bad service without a rebuild. This note designs a versioned, identifiable, reversible release process — applied the same way on Railway now and AWS ECS later — without adopting a platform (ArgoCD, Jenkins, CodeDeploy) this team doesn't need.

Everything below is either a fact verified directly against the three repos and live Railway/GitHub state (cited with file:line), a recommendation with its trade-offs stated, or — where marked — a decision that has since been made. See Decisions made and Still open for the current state of each.

What changed across revisions

See Changelog at the bottom for what was added at each revision and why — several follow-up questions surfaced real design gaps, not just clarifications.

Current process vs. proposed process

A visual summary before the detail below — the current gaps on top, the proposed tag-first process underneath, and how each gap closes at the bottom. Phase status shown is as of this revision; see Phased rollout and the Changelog for exact PR references.

Current state, verified

This section is a dated pre-implementation snapshot, not live state

Everything below was true when first written, before Phase 1 shipped — kept as the "why this design was needed" record, not maintained as an ongoing status page. It is now significantly out of date: real tags exist for all 14 services (see Tagging scheme), builds fire on tag push (not only nightly cron — the nightly schedule: trigger was itself removed 2026-09-06, see the v0.20 changelog entry), and the manifest-driven pipeline described later in this doc is live and fleet-verified. For current status, see the top-of-page Status line and the Changelog's newest entries, not this section.

Zero git tags and zero GitHub releases exist in blitz, blitz-ui, or fin-infra. Clean slate — no legacy tag format to preserve compatibility with.

Nothing builds on merge. Neither blitz nor blitz-ui has a push trigger. Images are only produced by a nightly cron (~02:30 IST) or manual dispatch. Merge-to-deployable latency is up to 24 hours today.

Every image is pinned to :latest, in three places

These are the same services declared three times across three IaC files — not three different sets of services. Railway's 14 (12 backend + 2 frontend) are what's actually live today; the AWS ECS declarations are Terraform only, not yet deployed, and currently describe fewer services than Railway does.

Location	Services declared	Tag
railway/backend/infrastructure.ts	12 (live)	:latest
railway/frontend/infrastructure.ts	2 (live)	:latest
aws/modules/ecs/services.tf (lines 128, 170)	10 (not yet deployed)	:latest

AWS IaC is missing two services — dated, not hypothetical

The AWS ECS module (aws/modules/ecs/services.tf) declares 10 backend services — it's missing blitz-apimgmt and blitz-bapiproxy, which Railway already runs. This isn't a design choice, it's a timing gap: the ECS module was written 2026-07-02. blitz-apimgmt was added to Railway's IaC on 2026-07-18 (16 days later), and blitz-bapiproxy on 2026-08-05 (over a month later). The AWS module simply predates both services and needs the same two service blocks ported before AWS go-live — independent of, and prior to, the release-management work below.

Editing the tag in infrastructure.ts does nothing today. provision.ts short-circuits on any service that already exists — there is no image-converge step, unlike volumes, limits, and registry creds, which are converged for existing services.

The primitives already exist, unused

The three things a release system needs are already present in the Railway API: ServiceInstanceUpdateInput.source.image is settable post-create through the client's existing generic passthrough (no new client code needed); Deployment.meta.imageDigest is readable, so the actually-running digest can be verified, not just assumed; and Deployment.canRollback / snapshotId is a native rollback primitive nothing currently uses.

Terraform was already designed not to be the deployer. Every aws_ecs_service resource in aws/modules/ecs/services.tf carries lifecycle { ignore_changes = [desired_count, task_definition] } (lines 220, 242, 264). The deploy mechanism has to live outside terraform apply — a manifest-driven deploy script fits this without fighting the IaC.

ECS pulls from GHCR, not ECR (aws/modules/ecs/services.tf:130, repositoryCredentials set against GHCR) — consistent with the documented "GHCR, no ECR migration" decision. This matters because ECR supports pattern-based tag immutability and GHCR does not — see Tagging scheme.

No release orchestration exists, and what's closest is broken.railway/scripts/release-build.ts and .github/workflows/release-staging.yml ("one-click staging release orchestrator") have never been run — zero workflow runs via the GitHub API. Two live defects would surface immediately on first use: the dispatch payload sends an image_type input that no longer exists on two of its three targets (would 422), and the single-service staleness gate (blitz-api stands in for all 12) is the exact class of bug already fixed in blitz's own build workflow but never ported here.

Also broken

release-staging.yml:120 gates its notify job on failure/cancelled, making the success-message branch on line 124 unreachable — the orchestrator, if run, always reports failure regardless of outcome.

Health-check coverage is thin, and it's an organic gap, not a deferred decision. Only 1 of 12 blitz services has a /health route (blitz-api), and it's mounted at /api/health while every reference to it (fin-infra IaC, the rollback runbook) says /health — so the declared path 404s. bapiproxy maps zero routes at all. Neither frontend's nginx config has a health endpoint, and its SPA fallback returns HTTP 200 for any unmatched path — so a naive health probe against a frontend proves only that nginx is serving files, nothing about the app inside.

No GitHub issue tracks this in blitz — there's no shared health-check plugin in libs/appbase for a new service to inherit, nothing in CI or review enforces one, and Railway's healthcheckPath field is fetched by the drift check and then never compared against anything, so a missing or wrong path has never once turned a check red. Worth its own tracked issue (mirroring fin-infra#35's shape) rather than fixing silently inline.

Resolved: runtime version visibility, for all 12 backend services

Distinct from the /health gap above — this is about identity ("which build is actually running"), not liveness ("is the process up"). First surfaced 2026-09-06 by the manifest-driven E2E test: the tag-derived version correctly reached the pushed image's org.opencontainers.image.version OCI label and the manifest — so an operator could always confirm what should be running — but for several services nothing actually read it back from inside the running container.

The initial scoping of this gap (4 pure-worker services) turned out to be wrong on a recheck: excelrw was initially listed as lacking a version endpoint, but it already had one (VersionController.cs, found on a corrected search) — the real gap was narrower in one place and wider in another:

• emailprocessor, wfw, wfwpdf (Bun workers, no HTTP surface — confirmed: no controllers/ directory, no Elysia app) had no way at all to surface version. Fixed: each now logs version.json's value in its existing startup log line — the practical minimum signal for a worker with no route to add one to.
• classicml (Python — a third runtime exception alongside the two .NET services, per fin-infra's CLAUDE.md service table; a ML Temporal worker using uv/scikit-learn/ FLAML, not Bun — classical-ML tooling with no real TypeScript equivalent, the same "right tool's ecosystem" reasoning as the .NET pair) turned out to have a deeper gap than the other three: its Dockerfile.py.service never had the SERVICE_VERSION → version.json bake wired in at all, unlike every other service's Dockerfile. Fixed: added the same bake pattern, read at runtime directly by main.py (no bundling step exists for Python to inline it at, unlike bun build's JSON inlining for the Bun services).
• bapiproxy (.NET, genuinely HTTP-serving) does not share excelrw's VersionController — separate .csproj, no project reference — so despite having an HTTP surface, it had the same gap as the Bun workers. Fixed: a new VersionController.cs mirroring excelrw's, exposing GET /bapi/version via the existing UsePathBase("/bapi").
• excelrw needed no fix — it already had GET /version reading AssemblyInformationalVersionAttribute, which the existing dotnet publish -p:InformationalVersion= bake already populates correctly.

Landed in blitz#1439. Not independently compile/build-verified for the .NET controller (no local dotnet SDK in the session that made the change) — worth a live check once a tagged build actually runs for bapiproxy.

Why not ArgoCD, Jenkins, or CodeDeploy

ArgoCD / Flux exist to solve one specific problem: continuously reconciling live Kubernetes cluster state against Git. ECS has no cluster-level state to drift — a deploy is "update the task definition," full stop. There is nothing for a sync controller to reconcile. Relevant only if EKS enters the picture for other reasons; not a version-pinning solution in its own right.

Jenkins / Spinnaker / Harness are orchestration platforms with real standing costs (their own HA, upgrades, RBAC). Redundant when GitHub Actions already runs CI and can call aws ecs update-service or the Railway GraphQL API directly.

AWS CodeDeploy is not deprecated, but AWS's own ECS documentation now opens with: "We recommend that you use the Amazon ECS blue/green deployment." CodeDeploy also cannot use the deployment circuit breaker at all — that mechanism is only supported on the native ECS deployment controller, rolling or blue/green. Native blue/green (GA July 2025) plus circuit breaker plus CloudWatch-alarm rollback covers the same ground CodeDeploy did, without a second control plane, an AppSpec file, or the ~25 extra IAM actions CodeDeploy needs.

What the circuit breaker actually is

ECS's built-in automatic-rollback mechanism for a rolling deployment — a counter, not a full canary system. When a new task-definition revision deploys, ECS watches new tasks in two stages: first, whether they fail to reach RUNNING (crash-on-boot); once at least one is running, it switches to watching health-check failures (ALB target group, Cloud Map, or the container's own HEALTHCHECK). Each failure increments a counter; once it crosses a threshold (default ~50% of desired count, clamped 3–200), ECS marks the deployment FAILED and, if rollback: true, automatically redeploys the last task-definition revision that reached a stable COMPLETED state — no CloudWatch, no CodeDeploy, no manual step. Its blind spot: it cannot see a task that starts, passes its health check, and then serves errors under real traffic — that needs the CloudWatch-alarm pairing described below. And because it's a property of ECS's own scheduler loop, it has nothing to hook into once CODE_DEPLOY replaces that loop with its own orchestration — hence the "unsupported" line above.

What ArgoCD popularized — Git as the source of truth for deployed versions, drift detected and corrected against it — is exactly right. It just doesn't need a Kubernetes-shaped tool underneath it. The release manifest (below) plus a thin converge script per platform is that pattern, hand-built to match a runtime that isn't Kubernetes.

Three-tier environment model

A correction from an earlier draft, not a refinement — treating this as Railway-staging-vs-AWS-prod understates what "preprod" needs to mean.

Railway today functions more like dev than staging — fast iteration, low ceremony. A pre-prod environment meant to gate production needs to be infra-identical to prod, which Railway (a different platform entirely from ECS) structurally cannot be. That gate has to exist on AWS itself.

"Low ceremony" does not mean :latest

An earlier version of this doc allowed dev to stay on a build "up to 24h old," implicitly via :latest. That's now corrected — see Tagging scheme and The release manifest: every tier, including dev, deploys an explicit version tag, never :latest, and every tier's manifest PR still goes through human review before merge (decided under Decisions made — no environment gets a lighter approval path). What makes dev fast isn't a looser artifact or a looser process — it's that tagging and reviewing a one-service manifest bump is quick work, not that either step is skipped.

The AWS preprod environment already exists — it needs finishing, not creating

aws/environments/ already has both preprod and production root modules. preprod has been applied (.terraform.lock.hcl present, per the fin-infra CLAUDE.md). production exists but is thinner — a diff shows it's missing the rds_autostop and elasticache modules that preprod has, among smaller differences (log retention, KMS output field names). It reads as scaffolded and never brought current with preprod's shape, not as a deliberate reduction.

Tier	Environment	Purpose	Manifest file
Dev	Railway	Fast iteration; explicit version tag, same as every other tier	release.json (under railway/environments/staging/)
Pre-prod	AWS preprod	Infra-identical to prod (same ECS/RDS/networking shape); gates promotion	aws-preprod.json
Prod	AWS production	Customer-facing	aws-production.json

This reshapes two things: the manifest design needs three files, not two, and the promotion chain becomes dev → preprod → prod. It does not mean Railway/dev gets a looser deploy artifact than AWS — every tier deploys by version tag (see Tagging scheme). What genuinely differs by tier is the rollback mechanism's maturity: Railway/dev's is a soft, scripted health-poll stopgap since no native circuit breaker exists there, while AWS preprod gets the full rigor (ECS circuit breaker, CloudWatch alarms) proven before anything reaches production, since preprod's entire purpose is to be a faithful rehearsal.

Prerequisite, not part of this project

Bringing aws/environments/production up to parity with preprod's module set (adding rds_autostop, elasticache, etc.) and porting the two missing services into aws/modules/ecs/services.tf are both small, mechanical Terraform PRs — this is Phase 4 below, sequenced before the AWS deploy-path work rather than folded into it.

Three concepts, kept separate on purpose

Manifest, tagging, and release identifier solve different problems and shouldn't collapse into one mechanism.

Concept	Answers	Lives in
Image tag	"What is blitz-api:1.4.0?" — identifies one service's build	GHCR
Release manifest	"What's running in an environment right now?" — every service's version, together	Git
Release identifier	A name for one point in the manifest's history	Not a separate artifact — it's the commit/PR that changed the manifest

Per-service semver alone answers "what is this build" but not "what is the product." The manifest is the join across independently-versioned services that makes the second question answerable.

Branching strategy

An earlier draft specified what governs what's deployed (the manifest) but not what governs what's buildable per environment — a separate axis that needs its own answer.

Option A — no environment branches; the manifest does all the work (recommended).main stays the only long-lived branch. Any commit on it can be tagged (api-v1.4.0) once built and verified. Promotion from dev → preprod → prod is purely "point the next tier's manifest at a version already proven in the previous tier's manifest" — the underlying image is identical either way, built once, promoted by reference, never rebuilt. This avoids ever asking "does prod's branch have this fix or not," since the manifest already answers that, and sidesteps rebuilding the same commit under a new tag (which would defeat immutable versioning — see below).

Option B — a staging branch as a promotion gate. Matches the move already noted as planned for unrelated reasons (three places currently hardcode main as the build source). Here, main→staging merge is itself the trigger that builds+tags, and a further branch gates the prod manifest update. Familiar Git-flow shape, but it risks two parallel sources of truth — which branch a tier is at, versus what its manifest says is deployed — that can drift from each other, which is precisely what a single-manifest design is meant to avoid.

Decided: Option A

Branches control what's mergeable; the manifest alone controls what's deployed. No environment branches. This also settles the branching half of the already-planned staging-branch move noted above — that move stays worth doing for its original reasons (the hardcoded-main build-source issue), but it does not become a promotion gate.

This governs environment promotion only. A separate, orthogonal question — patching one service's already-cut version without dragging in unrelated main churn from other services — does use short-lived branches, per service, not per environment. See Patching one service without dragging in the rest of main.

Tagging scheme

Format: per-service, independent semver, scoped at the deployable-app level: <service>-vX.Y.Z — e.g. api-v1.4.0, excelrw-v0.9.2, ui-v2.1.0. Cheap to create, cheap to query (git tag -l "api-*"), and a natural fit with per-service GitHub Releases if changelogs are wanted later.

Already half-built

All three build workflows — blitz's _build-push-service.yml:88-93 and blitz-ui's two UI workflows — already contain a refs/tags/* branch that derives a GHCR tag from a pushed git tag. It has never fired, because nothing has a tags: trigger and no tag has ever been pushed. Activating this is one line per workflow (on: push: tags: ['<service>-v*.*.*']), not new logic.

Version numbering — how X.Y.Z is decided, and the one hard rule

The version string is not auto-generated — a human picks it by hand at tag-creation time. The build workflow's derivation (_build-push-service.yml:96) is purely mechanical string surgery (emailprocessor-v0.1.1 → strip the emailprocessor-v prefix → 0.1.1); it does not validate the number against semver rules, prior tags, or anything else. That validation is what this section (and the CI check below) exist to add.

The one rule that can never be broken: no two builds of the same service ever share a version number, and a new tag is never a step backward. This is mostly free — git and GitHub already refuse to push a tag name that already exists, and tags are treated as immutable (never delete/retag a shipped release) — but nothing stops a new, lower or equal number being tagged after a higher one already shipped (e.g. accidentally reusing v0.1.1 weeks after v0.5.0 went out). See Version monotonicity check below for how that's actually enforced.

Bump rule (operational semver, not API semver). Standard semver (semver.org) assumes a declared public API most of these services don't have, so "breaking" is redefined here as operational, not wire-format:

• MAJOR — deploying this build requires something beyond "just deploy": a non-backward-compatible migration, a renamed env var/queue another service depends on, a required manual step, or a change you know is not safely rollback-able.
• MINOR — new feature/endpoint/workflow capability; additive migration; safe to roll forward without side effects.
• PATCH — bug fix, perf, dependency bump, logging — no behavior contract change.

Every service starts at 0.1.0 and versions independently — bumping wfw never touches emailprocessor's number, and there is no repo-wide version. Don't rush to 1.0.0; it's a signal a service has a stable-enough contract to matter, not a milestone owed to anyone by a deadline.

Deliberately not automated: semantic-release / changesets. Both are built for publishing packages to a registry, not container images, and semantic-release additionally requires disciplined Conventional Commits across every contributor — commit messages are a poor proxy for operational blast radius, which is what the bump rule above actually cares about. A human choosing the number, backed by the CI monotonicity check, gives the same real guarantee (no overlap, ever) for far less machinery — the right tradeoff for a small team that explicitly wants this simple.

Version monotonicity check

version-monotonicity-check.yml (reusable, in fin-infra, called from blitz and blitz-ui via uses: finaisse-org/fin-infra/.github/workflows/version-monotonicity-check.yml@main — same cross-repo pattern and Actions-access requirement as cherry-pick-ancestry-check.yml/sync-finboard.yml) runs on every <service>-vX.Y.Z tag push. It parses the service name and version out of the tag, lists every other existing <service>-v* tag, and fails the push if the new version is not strictly higher (compared with sort -V, so 0.10.0 > 0.9.0 is handled correctly, not lexicographically). The first tag ever cut for a service always passes — there's nothing to compare against.

This does not block rollback

Rolling back means redeploying an already-tagged older version via the release manifest (deploy.ts --from-manifest) — that never creates a new git tag, so this check never runs for it and cannot block it. The check only fires when a new tag is pushed, and only rejects a number that's ≤ an existing one for that service. "Redeploy an old build" and "cut a tag that goes backward" are different operations; only the second one this check cares about.

What this deliberately does not check: whether the bump size (patch/minor/major) actually matches what changed — that's the human judgment call in the bump rule above, not something CI can verify from a version number alone.

GitHub limitation: never push more than 3 tags in one git push

Discovered for real 2026-09-06 while seeding the first v0.1.0 tag for all remaining 11 backend services in a single git push origin tag1 tag2 ... tag11: every tag landed correctly on the remote (confirmed via git ls-remote and GitHub's own tags API) and the ServiceTagProtection ruleset had no issue with it — but zero on: push: tags: workflows fired for any of them, including Build and Push blitz-* and this repo's own new Version Monotonicity Check. blitz-ui's equivalent batch of 2 tags in the same session fired correctly, which was the clue: this is a documented GitHub Actions limitation, not a bug in this repo's config — GitHub silently ignores every tag-push trigger if more than 3 tags are pushed in a single push operation. No error, no warning; the tags simply exist with no corresponding workflow run, which looks identical to "the workflow is misconfigured" from the outside (and cost real time in this session diagnosing config, rulesets, and Actions health before this was found).

The fix: push tags one at a time (git push origin <tag>, once per tag), never git push origin <tag1> <tag2> ... or git push --tags when cutting more than 3 in one sitting. Confirmed working: deleting and individually re-pushing each of the 11 stuck tags triggered every build and check correctly. This has a real, permanent cost implication too — every tag push runs Version Monotonicity Check as its own separate ~20s job, billed at GitHub's per-job 1-minute rounding-up minimum regardless of actual runtime, on top of the build workflow's own minutes; normal usage (one service tagged at a time) keeps this small, but a repeat of a multi-service bootstrap batch will hit this limit again if pushed all at once.

When to tag, and why not just tag main's current tip

A naive version of this — "tag whatever main is at when you're ready to ship" — breaks under normal development, because main doesn't stop moving while you're testing. If a build is deployed to Railway/dev, tested there, and only then tagged, the tag would point at main's tip at tag time, not the commit that was actually built and validated — those are two different commits the moment even one more PR has merged in between. The tag would misrepresent what was tested.

Decided: deploy is tag-first everywhere, :latest is never in the deploy path

The fix is to invert the order, on every tier including dev — there is no environment that deploys :latest:

1. Pick the exact commit you want to test or ship (usually main's tip at that moment, but it's any commit SHA — tags point at a specific commit, not a moving branch).
2. Tag it directly: git tag api-v1.4.0 <that-commit-sha>, git push origin api-v1.4.0.
3. The tag push triggers the build (see above) — this is the only build that commit ever gets. It produces one immutable artifact: ghcr.io/finaisse-org/blitz-api:1.4.0.
4. Deploy that tag to Railway/dev — never :latest. Whatever you observe while testing is provably api-v1.4.0, because nothing else was ever deployed.
5. If it's good, the same already-built image is promoted to preprod → production by reference via the manifest. No rebuild, ever, for a version that's already been cut — promotion only ever changes which environment's manifest points at an existing, already-tested image.
6. If it's bad, fix forward and cut api-v1.4.1. api-v1.4.0 is never mutated or re-pointed at different content.

This removes the "which SHA is actually running" question entirely — there's no post-hoc lookup against Railway's or ECS's deployment metadata to figure out what shipped, because the tag was chosen before the build, not inferred after it. The image's digest (used in the manifest, see below) is likewise known the moment the tagged build finishes — read directly off GHCR's push response — not derived by asking the runtime what it happens to be running.

The practical cost: testing a change on Railway/dev now requires a tag+push+build round trip (a few minutes) rather than waiting for the next nightly :latest build or a manual dispatch. Given the nightly cron is already up to 24h stale today, this is a net speed improvement for anyone actively iterating, not a slowdown — and tagging is cheap (one git command), so the ceremony is minimal.

Alternatives considered:

Approach	Pros	Cons	Why not chosen
Tag-first (chosen) — tag an exact commit, build only that, deploy only that tag	Tag always describes exactly what was built and tested; no post-hoc lookup ever needed	A few minutes' round trip before a change is deployable at all	—
Tag-after — deploy main's tip (or :latest), test it, tag it retroactively once it looks good	Zero round-trip latency to first deploy — test immediately	The tag can misdescribe what was tested the moment any other commit lands on main in between build and tag; this is precisely today's :latest problem, just renamed	Directly reintroduces the "what's actually running" gap this whole design exists to close
Keep :latest for dev only, tag-first for preprod/production	Dev stays as fast as today; only the tiers that matter for audit get rigor	Reintroduces exactly the two-tier asymmetry an earlier draft of this doc allowed and then corrected (see the warning above) — dev becomes the one place nobody can answer "what's running," which is also where most incidents are first noticed	Rejected under Three-tier environment model: no environment gets a lighter artifact discipline, only a faster review cycle

What tagging alone doesn't give you. GHCR has no tag immutability feature, in any form — confirmed absent via an unanswered, uncommitted GitHub community feature request, not merely undocumented. A pushed v1.4.0 tag can be silently overwritten later by anyone with push access. ECR does support pattern-based immutability (IMMUTABLE_WITH_EXCLUSION, GA July 2025 — immutable v1.2.3, mutable :latest, in one repo), but that's moot while ECS pulls from GHCR.

The mitigation is digest pinning, which fin-infra#35 (open) already asks for. The manifest carries the digest, not just the tag — see The release manifest.

What "pattern-based immutability" on ECR actually means, concretely

An ECR repository's imageTagMutability setting used to be all-or-nothing — MUTABLE (any tag overwritable) or IMMUTABLE (once pushed, that tag can never be pushed again, including :latest, which breaks normal CI). AWS added IMMUTABLE_WITH_EXCLUSION in July 2025: the repo is immutable except for tags matching an exclusion wildcard. Set the filter to latest and you get, verbatim from AWS's own example: v1.4.0 can never be silently overwritten once pushed, but :latest stays freely re-pushable. GHCR has no equivalent setting at all — every tag on every GHCR package is always mutable — which is exactly why the manifest records digest rather than trusting the tag alone.

GHCR growth & retention

A real, already-existing problem — not a hypothetical one introduced by adding version tags.

Confirmed during the audit

blitz-api already has 482 image versions in GHCR, blitz-ui 370, blitz-wfw145, blitz-mgmtui 107 — accumulated purely from the current SHA-tag-per-build scheme, with zero retention policy anywhere in any of the three repos. Adding version tags doesn't change the growth rate (one more tag on the same image, not a new image) but doesn't fix this either.

Rollback does genuinely require older images to remain available — but that means a deliberate retention policy, not today's default of keeping everything forever. The standard shape: retain every version referenced by any manifest still considered live (current + last N releases per environment, so a rollback target always exists); garbage-collect untagged/SHA-only builds more aggressively, since they're not addressable by version anyway; and run a scheduled cleanup (actions/delete-package-versions, or a script against GitHub's package API) that treats "still referenced in a manifest" as the retention criterion.

This is a natural fit for Phase 2 below — a manifest that lists every still-referenced version is exactly what tells a retention job what's safe to delete, so the two are best built together rather than sequentially.

The release manifest

One JSON file per environment, git-tracked. This is the artifact that answers "what version of the product is running."

// railway/environments/staging/release.json
{
  "release": "2026-08-31-1",
  "environment": "railway-staging",
  "committedAt": "2026-08-31T10:12:00+05:30",
  "releasedBy": "sekhar@finaisse.com",
  "services": {
    "blitz-api": {
      "version": "1.4.0",
      "image": "ghcr.io/finaisse-org/blitz-api:1.4.0",
      "digest": "sha256:246a5716183454b22594d7bbd226333671b768e00ce7a8594a2741af2f5133a"
    },
    "blitz-wfw": {
      "version": "1.1.2",
      "image": "ghcr.io/finaisse-org/blitz-wfw:1.1.2",
      "digest": "sha256:9a8b7c6..."
    }
    // ...every deployed service, always all of them, even ones unchanged this release
  }
}

Digest, not just tag or SHA — this is the detail that matters most. The digest is known before any deployment happens: it's read straight off the tagged build's own push output the moment <service>-vX.Y.Z finishes building (GHCR returns the digest of what it just pushed) — not derived after the fact by asking Railway or ECS what they happen to be running. That ordering matters: deriving the digest post-deploy is exactly the "figure out what shipped after the fact" pattern this design replaces (see Tagging scheme for why deploys are tag-first, never :latest-then-inspect). Railway's Deployment.meta.imageDigest and ECS's versionConsistency parameter remain useful as an independent verification signal — confirming the deployed digest matches what the manifest expects — but they are not where the manifest's digest value comes from.

Where it gets committed: into fin-infra itself, alongside the IaC that reads it — railway/environments/staging/release.json for Railway, and (once AWS deploy lands) aws/environments/preprod/release.json and aws/environments/production/release.json, matching each tier's existing environment-directory convention rather than a separate top-level release-manifests/ folder. This repo is where every consumer (the deploy scripts) and the only writer (a deploy workflow) already live — keeping the manifest here means one commit changes both "what the IaC declares" and "what's actually deployed," and git log on that one file is the complete release history with no cross-repo correlation needed.

Update discipline — decided: a deploy workflow is the only thing that ever proposes a manifest change; nobody hand-edits the JSON, in any environment. But the workflow never commits directly — every manifest change lands via a PR that a human merges, the same process in all three environments (no PR-gate asymmetry between dev and production). This is a real checkpoint, not ceremony around a rubber stamp: it is the one place a human sees the full diff of what's about to become "the record of what's deployed" before it's final.

The manifest must never assert more than what actually succeeded

Do not write the manifest as a statement of intent ("here's what we're about to deploy") and commit it before, or blindly after, the deploy step. That ordering has a real failure mode: if a 10-service deploy fails partway — say 8 succeed, 2 don't — a manifest written from the pre-planned diff (or from a bare "the job exited 0" check) would claim all 10 are at the new version when only 8 actually are. That's not a cosmetic bug; it's the exact false-positive shape this design exists to prevent (see Current state, verified — Railway's fleet already drifted from :latest invisibly once, with no manifest in the loop at all).

The rule: each service's deploy is verified individually (the health-check / circuit-breaker / DescribeServiceDeployments gate already described below) before that service's entry is written into the manifest diff. The workflow's final step opens the PR built from what was confirmed successful, not from what was planned — if 8 of 10 succeed, the PR proposes exactly those 8 changed entries, the other 2 stay at their previous value, and the workflow's own run exits non-zero so the partial failure is visible on the run itself, not only inferable from a smaller-than-expected diff. The manifest is a record of a confirmed state, never a plan.

Answering "what's running right now": cat aws/environments/production/release.json, or git show <commit>:aws/environments/production/release.json for any past point in time — and because of the rule above, that answer is always something that was actually verified running, not merely requested. A small addition worth having once all three tiers run continuously: a scheduled job that diffs adjacent tiers' manifests and reports what's pending promotion.

A narrower, earlier assertion: Railway staging's tag-time manifest automation

The rule above governs a manifest entry asserting a deploy succeeded. Railway staging's tag-time automation (update-manifest-from-tag.yml, shipped 2026-09-06 — see Version monotonicity check for the sibling piece) writes a manifest PR earlier in the pipeline than that: right after a tag's build succeeds and its image is independently re-verified in GHCR, not after any deploy. This is a deliberate, narrower assertion — the PR claims only "this exact image was built and verifiably exists in GHCR," never "this image is running." A merged manifest PR from this automation is not by itself proof staging is running that version — it becomes true only once the merge-triggered deploy has actually run against it. Don't read "the manifest PR merged" as "the service redeployed" for Railway staging; that stronger claim still requires the deploy step, exactly as it always has.

(2026-09-13: the merge now triggers that deploy automatically, so the gap is much shorter — but the distinction stands. A merge means the deploy was started, not that it succeeded; bun run manifest:drift is still the only thing that proves Railway is running it.)

Historical rule — resolved 2026-09-13 by the merge trigger, kept for the incident record

deploy-from-manifest-diff.ts diffs the manifest's current committed content against its immediately previous commit only (HEAD~1 vs HEAD) — not against "everything since the last actual deploy." If two manifest-changing PRs merge back-to-back and the deploy workflow only runs once, after both, the diff only reflects the SECOND PR's change; the FIRST PR's change is silently invisible to that run, forever, unless caught by hand.

Hit for real 2026-09-06: tagging all 14 services in one session, PR #130 (agents) merged first, then PR #143 (the other 12 services, combined into one PR after #130's merge caused the original 12 individual PRs to conflict on the shared release/committedAt lines) merged second. Deploy from Manifest Diff was run once, after both had merged — it correctly redeployed all 12 services in #143, but agents was never included in any deploy run and kept running stale :latest for the rest of the night, discovered only by chance via a screenshot of Railway's dashboard showing "Service is offline" / no active deployment. Fixed by hand via a direct deploy-staging.yml --from-manifest agents dispatch, independently re-verified against Deployment.meta.imageDigest.

This rule no longer applies, and the manual workflow it names is deprecated. The code fix it was waiting for arrived as the pull_request: closed trigger (2026-09-13): a deploy now fires on each manifest merge, so every run's HEAD~1 is that PR's own parent and the batching case cannot arise. Since 2026-09-17 the five workflows that mutate Railway staging also share one railway-staging-mutation concurrency group, so two merges landing seconds apart queue rather than race.

What still needs watching: if one merge's deploy fails and another manifest PR merges on top, the failed change appears in no later diff — the same silent-skip shape, reached a different way. That is what the daily Drift Check exists to catch; bun run manifest:drift is the on-demand version. Do not re-add the manual rule: run the drift check instead.

Superseded in mechanism, 2026-09-13 — but the constraint still holds. The manual step is gone: Release Manifest and Deploy now triggers on any merged PR touching release.json (fin-infra#219), so a merge deploys itself and cannot be forgotten. The HEAD~1 window is unchanged, so merging two manifest PRs back-to-back still loses the first — concurrent runs are serialised by the workflow's concurrency group rather than raced, but each still diffs only its own HEAD~1. The rule is therefore now: merge one manifest PR, let its deploy finish, then merge the next. Note a squash-merged combined PR lands as one commit, so an aggregate PR covering 14 services diffs correctly as 14.

A code-level fix (recording a lastDeployedSha on the manifest itself, diffing from there instead of always HEAD~1) was scoped but deliberately NOT built: main's branch protection requires every commit to land via a reviewed PR, and there is no way to grant an automated write like this bypass access without granting that same actor blanket, un-reviewed push access to main for anything — not just this one safe, deterministic field. Given the underlying bug is fully avoidable by discipline alone (see the rule above), that trade was judged not worth it. Revisit only if the batching temptation proves too easy to forget in practice.

Railway deploy path

Extend railway/scripts/deploy.ts to read the manifest and, per service, do two calls:

await client.updateServiceInstance(environmentId, serviceId, {
  source: { image: manifestEntry.image },  // config, staged
});
await client.deployService({ environmentId, serviceId }); // resolves it now — see below

The image lives on the service instance and lands on the next deployment — like other serviceInstanceUpdate fields. This is not a per-deploy parameter to Railway's API; it's a persistent property you set, then trigger — but which trigger matters; see the next callout for why this used to say redeployService.

Critical bug found and fixed 2026-09-06: redeployService silently ignored the image change

This section originally paired updateServiceInstance with client.redeployService(...), on the assumption that any redeploy re-resolves whatever source.image is currently configured. That assumption was wrong, and the manifest-driven deploy path had never once actually delivered a tagged image, for any service, until this was found and fixed.

Verified empirically while confirming the first live E2E test: emailprocessor-v0.1.1's manifest entry converged correctly (source.image genuinely read the tagged image), the deploy reported SUCCESS, and the container was even confirmed doing real work (IMAP polling) — but the actual running deployment was still ghcr.io/.../blitz-emailprocessor:latest, caught only because a screenshot of Railway's dashboard showed the wrong tag on the ACTIVE deployment card. Querying Deployment.meta.image directly confirmed this on 6 consecutive real deployments going back to 2026-08-30 — predating this session, and predating Phase 2 entirely. serviceInstanceRedeploy restarts the previous build; it does not read source fresh. Schema introspection found the real fix: serviceInstanceDeploy (a separate mutation, commitSha/latestCommit args suggest it's the "resolve source and deploy" operation Railway's git-connected services use) does read source.image fresh — confirmed by triggering it directly and watching Deployment.meta.image match.

Fixed in railway-client.ts (deployService, wrapping serviceInstanceDeploy) and deploy.ts (calls deployService only when this run just converged the image via the manifest; a plain redeploy/scale-up with no image change stays on redeployService, since there's no evidence it's wrong for that unrelated, image-unchanged case). Re-verified after the fix: the same emailprocessor-v0.1.1 deploy now shows Deployment.meta.image correctly matching, confirmed independently via both a live GraphQL query and the Railway dashboard.

Lesson for this doc's own discipline: "the config converged" and "the right image is running" are different claims — this section, and an earlier summary in this same investigation, asserted the second without directly verifying it. The only reliable check is reading back Deployment.meta.image (or the dashboard) after a real deploy, not trusting a SUCCESS status or a converge call's return value.

Closing the "is it actually running" gap. check-image-drift.ts currently compares GHCR's :latest SHA against the source repo's main tip — it never queries Railway, and its own docstring says so. A live check during this investigation found all 14 app services running the previous build while :latest already pointed at a newer one — benign that time, invisible to every existing check. Once the manifest records digest, a companion check can join Deployment.meta.imageDigest against the manifest's expected digest and catch exactly this class of drift.

Phase 2.5 — one source of truth for image version (done)

Phase 2 minimal scope landed the manifest and an opt-in converge path (deploy.ts --from-manifest), but deliberately left infrastructure.ts untouched to keep that PR small. Auditing the shipped code afterward found two declarative sources for "what image should this service run," not one: infrastructure.ts hardcoded :latest for every service, and the manifest was consulted only when deploy.ts ran with --from-manifest — a plain deploy.ts <target> never looked at the manifest at all.

Decided and shipped: the manifest is the sole source, no live "is this a no-op" check

The first draft of this fix reached for comparing the manifest's entry against "what's currently declared" before converging — first against infrastructure.ts, then, once infrastructure.ts stopped carrying a real tag, against Railway's live service-instance state. Both are the wrong shape: deciding what should deploy by first asking the platform (or a stale local file) what it currently thinks is running is exactly the pattern this design exists to avoid — the version is decided once, at tag time, and the manifest is the only place that decision should need to be read from.

The fix: convergeServiceImage (railway/shared/manifest.ts) now sets a service's image to its manifest entry unconditionally whenever one exists — no comparison, no baseline, no "no-drift" skip branch. This costs nothing: updateServiceInstance is idempotent, so converging to a value a service already has is a harmless no-op API call. Removing the optimization removed the need for a second source of truth entirely.

Landed as:

1. railway/shared/manifest.ts — convergeServiceImage drops the declaredImage parameter and the drift comparison; converges unconditionally.
2. railway/scripts/apply-manifest-images.ts / deploy.ts — both drop their declaredImages() baseline (a Map built from infrastructure.ts) entirely; iterate the manifest's own service names instead.
3. railway/backend/infrastructure.ts / frontend/infrastructure.ts — image is now a bare, tagless GHCR reference (e.g. ghcr.io/finaisse-org/blitz-api, no :latest) — which repo a service pulls from, not which version. infrastructure.ts declares infrastructure only (ports, resources, env, topology); the manifest is the sole source for version.
4. provision.ts — the one place a tag is still needed: a brand-new service has no manifest entry to converge from at create time, so bootstrapImage() appends :latest explicitly, as a one-time value. Step 8 (manifest converge) runs immediately after creation in the same script run and repoints it to a real manifest entry if one exists.
5. check-image-drift.ts — needed no change: it only used .image to derive the bare image name (stripping any tag) for its GHCR-vs-main staleness check, which is unaffected by infrastructure.ts no longer carrying a tag.

No dependency on every service having a real manifest entry first — the "unset"/:latest placeholder entries converge exactly the same way real ones will, since there was never a comparison to trip over.

AWS ECS deploy path

Why this matters: ECS pulls from GHCR, not ECR

aws/modules/ecs/services.tf:130-131 already sets repositoryCredentials against GHCR, per the documented "GHCR, no ECR migration" decision. Three consequences: no registry-side tag-immutability guarantee is available on AWS either, since GHCR has none — the only real integrity guarantee on either platform is the digest pin in the manifest, which works identically cross-registry. Second, this is an ongoing operational dependency: each task definition needs a valid GHCR PAT in Secrets Manager to pull at all, and an expired PAT fails task launches the same way it's already bitten Railway pulls. Third, migrating to ECR later purely to gain tag immutability would be swimming against an explicit prior decision — not recommended unless that decision changes for other reasons.

Deployment strategy: native ECS, not CodeDeploy. Default every service to the ECS deployment controller with strategy: ROLLING, deployment circuit breaker on (enable=true, rollback=true), and CloudWatch alarms wired for rollback on 5xx rate / latency / CPU. Reserve strategy: BLUE_GREEN for the two public-facing services (the ALB-exposed API gateway and WS equivalents) where a bad rollout is customer-visible; leave Temporal workers on rolling, since blue/green's benefit (pre-production traffic validation) doesn't apply to a service with no inbound traffic to validate.

	Rolling + circuit breaker + alarms	Native blue/green	CodeDeploy
Circuit breaker support	Yes	Yes	No — unsupported controller
Extra Terraform surface	Minimal	2 target groups + listener rule + infra role, per service	App + deployment group + AppSpec; task-def handling moves outside TF
Capacity during deploy	Normal	Up to 2× (AWS's own documented caveat)	Normal
AWS's current recommendation	Default	Yes, for public-facing	"Existing customers only," per AWS docs

Deploy mechanism: register a new task-definition revision via aws-actions/amazon-ecs-deploy-task-definition, pointing at the manifest's pinned image (and digest, once written into the container definition), then update-service. This action has no blue/green or circuit-breaker–aware input — it just calls UpdateService, which is fine, since the strategy lives on the service definition in Terraform, not in the deploy call.

Confirmed reliability trap

Both the action's wait-for-service-stability input and the raw aws ecs wait services-stable CLI waiter use a stability condition that only checks "one deployment, runningCount == desiredCount." A deployment that failed the circuit breaker and rolled back converges to exactly that state — so the waiter can report success on a rollback, i.e. a failed deploy reads as green. This is the identical failure class this repo already documents for Railway's redeployService (a stale-deployment read passing instantly) — same shape, different platform. There's also an open, unresolved Terraform provider issue (#19519, since 2021) describing the same defect in wait_for_steady_state.

Fix: gate CI on DescribeServiceDeployments reaching SUCCESSFUL, failing on any ROLLBACK_*/STOPPED state — not on the stability waiter. This maps directly onto the same lesson already encoded in railway/shared/wait-for-deploy.ts; the AWS-side script should carry the same discipline, not reinvent it worse.

Blue/green — native, not built

Blue/green is a built-in ECS deployment strategy (GA July 2025), not a separate product and not something to construct — strategy: BLUE_GREEN on the same standard ECS deployment controller used for rolling deploys, not CodeDeploy. It's configured declaratively on the aws_ecs_service Terraform resource itself (Terraform provider support since v6.4.0): a second "alternate" target group, a bake time, optional lifecycle-hook pause points. AWS's own ALB traffic-shifting mechanics perform the actual swap.

The only real cost is the extra Terraform per service that opts in (a second target group + listener rule) and roughly double task capacity during the deploy window — which is exactly why the AWS deploy path above reserves it for the two public-facing services rather than enabling it fleet-wide.

Rollback design

Rollback in both environments is manifest-driven: revert the environment's manifest commit, redeploy from it. No rebuild — minutes, not hours.

Railway. No native circuit breaker exists here. Two layers: (1) Deployment.canRollback / snapshotId is a real, currently-unused Railway-native rollback primitive — worth wiring in directly rather than only reasoning in terms of redeploying an old tag. (2) A scripted post-deploy health-check poll with a timeout that, on failure, automatically re-deploys the manifest's previous entry for that service. This is a deliberate stopgap versus ECS's native mechanism, and should be labeled as such to the team.

Trap for this design specifically

Health-gated auto-rollback needs a real health signal. Today, 11 of 12 blitz services have no /health route, and both frontends' SPA fallback returns HTTP 200 for anything unmatched. A rollback trigger wired to those endpoints as they stand would never fire, or fire on the wrong signal. Health-route coverage is a prerequisite, not a parallel task.

AWS ECS. Enable the deployment circuit breaker (rolling back to the last COMPLETED deployment automatically) plus CloudWatch alarms for the class of failure the breaker can't see — a task that passes health checks but serves errors under real traffic. Manual override via StopServiceDeployment (1-click rollback, GA May 2025) or native blue/green's instant traffic-flip for the ALB-fronted services.

A documented sharp edge worth knowing before relying on this

The circuit breaker only rolls back to the most recent COMPLETED deployment. If there is no such deployment — e.g. the very first rollout, or two bad deployments in a row — it does not roll back; it stalls with no new tasks launched. A bad deploy immediately followed by another bad deploy has nowhere to fall back to.

Partial rollback: one service out of ten

"Revert to the previous manifest" doesn't survive contact with the most common real incident: one service is bad, the other nine are fine, and nobody wants to touch them.

The honest mechanics: a partial rollback is not reverting the whole manifest to a prior commit — it's authoring a new manifest state that's mostly-current with one service's entry pinned back to its previous {version, image, digest} (read straight from the prior release commit — git show <prev-commit>:release.json | jq '.services.recon'). The other nine entries stay untouched. Same order as any other deploy, per the update discipline above: redeploy that one service, verify it's healthy on the old version, then open the PR with just that one entry changed — never commit the intended rollback ahead of confirming it worked.

This is not a distinguished operation

Mechanically, a rollback and a forward deploy are the same action: edit the manifest, redeploy the affected service(s). What makes something "a rollback" is only that the version being pointed at happens to be older than the one it replaces. The tooling needs no separate rollback mode — the same script that promotes one service forward also rolls one service back, just by pointing at an older tag.

The sharper question is what this does to the release identifier. "Release 2026-08-31-1" named an all-ten-services combination; the moment one service is swapped, that name stops describing what's actually running.

Alternatives considered:

Approach	Pros	Cons	Why not chosen
Stable name + tracked variance (chosen)	Release identifier stays meaningful as "the combination that was actually signed off on" as a unit; one release name maps to one deliberate sign-off event	One more field to track (the variance note) alongside the name	—
Mint a new release identifier on every manifest change, including single-service rollbacks	Simple rule, no extra field — one manifest commit, one name, always	The release count inflates with every partial hotfix and stops distinguishing "we deliberately shipped a new combination" from "we patched one thing back" — the name stops being useful for exactly the audit question it exists to answer	Rejected: makes the release history noisier, not more precise, for this repo's actual usage pattern (frequent small single-service fixes against infrequent whole-set cuts)
No release identifier at all — manifest commit SHA is the only name	Zero extra bookkeeping; git log is already a complete history	Loses the human-readable "which named release is this" question this whole design is meant to answer — a commit SHA doesn't tell a reader what was deliberately signed off on without opening the diff	Rejected under Three concepts, kept separate: a release identifier answering "what's this point in history called" is one of the three concepts this design deliberately keeps distinct from the manifest and the tag

Decided: keep the release name stable, track the variance

New release identifiers are reserved for deliberate, whole-set promotions — a partial single-service rollback does not mint a new one. Instead it's tracked as a documented variance from the named release (e.g. "currently: release 2026-08-31-1, with recon patched to 1.3.1"). This keeps the release identifier meaningful as "the combination that was actually signed off on," at the cost of one more field to track — a small price against every partial hotfix inflating the release count and obscuring what real promotions happened.

Either way, the manifest file itself stays the ground truth for "what's running right now" — this only affects how the human-facing release name is assigned on top of it.

Patching one service without dragging in the rest of main

The section above assumes the patched image already exists somewhere, ready to point the manifest at. This section is the step before that: where does a single service's patch release actually come from, once main has moved on?

Concretely: all 12 services are cut together at v1.4.0, from the same commit. Two weeks later api needs a fix, but main's current tip also carries two weeks of unrelated merges — to api itself and to shared libs//packages/ code most services bundle. Tagging main's tip as api-v1.4.1 would build api from a tree that also contains all that unrelated work — not reproducible as "v1.4.0 plus just this fix."

A premise worth correcting before the decision below

An earlier revision of this section framed the choice as "trunk-based development, so no release branches" vs. "a release branch per service." That's a false choice — confirmed against Google's own account of its release process, which is trunk-based development's most-cited large-scale example:

"Most projects don't release directly from the mainline. Instead, we branch from the mainline at a specific revision... Bug fixes are submitted to the mainline and then cherry picked into the branch for inclusion in the release." — Google SRE Book, Release Engineering

"Trunk-based" describes where development happens — no long-lived feature branches. It does not mean releases are always cut from trunk's moving tip. A release branch, cut just-in-time and deleted shortly after, is the standard mechanism trunk-based development itself uses for exactly this problem — see trunkbaseddevelopment.com/branch-for-release, which documents this as "branch for release" (cut when needed, cherry-picks only, deleted once the release is superseded) and, distinctly, "retroactive branching" (Brad Appleton) — branching backward from an already-shipped tag specifically because a bug surfaced in production after trunk had moved on. That second name describes this exact scenario.

Decided: a retroactive, ephemeral release branch per patch — repo-wide, cut from the tag, deleted after

Cut one branch from the exact tag commit — only when a patch is actually needed for a version already shipped, never pre-emptively. It is repo-wide, not per-service: this repo's builds share one tree, one lockfile, and heavy cross-service library coupling (see the warning below), so "a branch containing only api's code at that commit" doesn't exist as a concept here — the branch is a pin on a point in history, not a subset of it. It receives no ongoing development; the only commits that land on it are cherry-picks of fixes already merged to main. It is deleted once that patched version is no longer live in any environment — its whole lifetime is meant to be short.

# When api needs its first post-v1.4.1 patch — not before:
git fetch origin
git checkout -b release/1.4.x api-v1.4.1   # branches from the TAG, not main's tip
git push origin release/1.4.x

# ...cherry-pick the already-merged main fix, then tag ONLY the service that changed:
git cherry-pick -x <main-commit-sha>
git tag api-v1.4.2
git push origin api-v1.4.2   # triggers build-push-api.yml — only api rebuilds

The fix always lands on main first — normal PR, normal review, tested against current code — then gets cherry-picked backward onto the release branch. Never the reverse: fixing directly on the release branch and hoping to remember to forward-port it to main later is exactly the failure mode this rule exists to prevent, and it's the same rule Google's SRE book states for the same reason — it's what makes deleting the branch safe later, since nothing unique to that fix lives only on it.

Exception: the bug may not reproduce on main at all

"Fix on main first" assumes the bug is a real, still-present defect there. After a month of unrelated churn, that's not guaranteed — three ways it can fail:

• Already fixed by coincidence — an unrelated refactor touched the same code path and the bug is simply gone on main. There is nothing to cherry-pick; check the bug actually reproduces on current main before assuming a backport is the right mechanism.
• The same logical fix looks different on main — surrounding code moved enough that cherry-pick conflict resolution produces a materially different diff on the release branch than what was reviewed on main. Still author on main first, but expect the release-branch version to diverge, and re-review the resolved conflict, not just the original PR.
• The code is gone entirely — replaced or removed on main. There is no fix to write against main, and this is no longer a backport — it's a standalone patch to old, no-longer-representative code.

For the third case (and only that case), the rule's escape hatch: an admin or designated release moderator may author the fix directly on the release branch, bypassing the normal PR-first flow. This maps onto the ReleaseBranchProtection ruleset (blitz/fin-infra, both repos) that enforces "no direct push, PR + 1 approval to merge" on release/** — its bypass_actors is OrganizationAdmin-only today, which is also the intended user of this exception. This is a deliberate, logged exception, not a loophole for convenience — reach for it only once it's confirmed there is genuinely nothing on main to fix.

Sharp edge specific to this repo: a repo-wide branch does not mean a repo-wide tag

Because the branch is repo-wide but tags stay per-service, only tag the service(s) whose fix was actually cherry-picked onto the branch. If api's fix is cherry-picked and then someone later tags recon-v1.4.2 from that same branch "while they're there," recon silently ships whatever else has landed on the branch by that point — most likely nothing yet, but this stops being true the moment two unrelated patches are ever cherry-picked onto the same branch before either is tagged. Discipline: cherry-pick one fix, tag that one service, and treat the branch as holding exactly the patches that have already been tagged from it — don't let unrelated pending fixes stack up on it. If two services genuinely need unrelated patches at the same time, prefer two separate branches (release/1.4.x-api, release/1.4.x-recon) over sharing one, precisely to keep this invariant easy to see.

Known gap: a second patch on an already-patched line

The examples above all assume release/1.4.x doesn't exist yet. In practice a service can need a second patch — api-v1.4.1 shipped, then api-v1.4.2 was cut from release/1.4.x, and now a third fix is needed. Two different states are possible, and they need different handling — neither is spelled out above:

• The branch still exists (likely: 1.4.2 may still be live somewhere, so it hasn't been deleted yet). This is the easy case: reuse it. Cherry-pick the new fix onto the same release/1.4.x, tag api-v1.4.3 from its new tip. Don't cut a second branch — the existing one already is the correct lineage, and this is exactly the "repo-wide branch, per-service tag" discipline above applied to a third round.
• The branch was already deleted (someone judged 1.4.2 no longer live anywhere and cleaned it up). It must be re-cut from api-v1.4.2 — the latest tag actually shipped on this line — never from the original api-v1.4.1. Re-cutting from 1.4.1 would silently drop 1.4.2's fix from the new branch's history.

The gap this repo hasn't closed: nothing verifies "no longer live anywhere" before a release branch gets deleted. The release manifest is the right source for that check — a version is safe to consider dead once no environment's release.json references it — but no script or CI gate wires this check into the deletion step today; it's a manual judgment call. Getting it wrong either way has a real cost: deleting release/1.4.x too early means re-cutting it later has to correctly identify the latest surviving tag (1.4.2, not 1.4.1) rather than assuming the original; deleting it believing 1.4.2 is dead when it's actually still deployed somewhere is worse — the next patch would be cut from the wrong ancestor entirely if someone re-derived the branch from the tag they assumed was current. Not yet built: a grep <version> across every environment's release.json check, ideally run automatically before any manual branch deletion, not just recommended in prose.

Why cherry-picking doesn't degrade into unmanageable toil over time. The worry with any cherry-pick-based workflow is real: after enough patches, manually tracking which main commit went where gets error-prone. The industry answer isn't "avoid cherry-picking" — it's automate the mechanical step, keep the review a human's job. Kubernetes does this with a Prow bot triggered by a cherry-pick-approved label; the widely-used generic equivalent is a label/comment-triggered GitHub Action such as korthout/backport-action: apply a label like backport release/1.4.x to the merged main PR, and the bot cherry-picks it onto that branch (with -x, leaving a (cherry picked from commit ...) trailer for a durable audit trail where the pick applies cleanly) and opens a PR for review — it never merges directly. This removes the manual git cherry-pick toil without removing the human judgment call on a genuine conflict.

What's enforceable in CI here, and what isn't

Done: blitz and fin-infra both now carry two repo-level rulesets alongside the org-level DefaultBranchProtection — ReleaseBranchProtection (target branch, scoped to refs/heads/release/**: requires a PR with ≥1 approval to merge, blocks force-push and deletion, bypass restricted to OrganizationAdmin) and ServiceTagProtection (target tag, scoped to refs/tags/*: blocks deletion and force-update of any existing tag, leaves tag creation open, same admin-only bypass). So a release/* branch is no longer force-pushable or deletable by a normal contributor, and an existing tag can no longer be silently deleted or re-pointed — closing the two gaps this callout originally flagged.

Also done: the ancestry check itself — cherry-pick-ancestry-check.yml, a reusable workflow in fin-infra (called cross-repo from blitz the same way as sync-finboard.yml/lockfile-guard.yml), triggers on any push to release/** and, for every new commit carrying a (cherry picked from commit <sha>) trailer, runs git merge-base --is-ancestor <sha> origin/main and fails the check if that claimed origin isn't actually reachable from main. A commit with no trailer is never flagged — this still cannot enforce "must be a cherry-pick," only "if it claims to be one, the claim is real." Verified against three cases locally before shipping: a genuine cherry-pick passes, a fabricated trailer (citing a SHA that was never on main) fails, and a plain commit with no trailer at all is silently skipped.

Also shipped: backport.yml (both repos) wires up korthout/backport-action — labeling a merged main PR backport release/1.4.x cherry-picks it (with -x, feeding the ancestry check above) onto that branch and opens a new PR for a human to review and merge; it never merges directly. Triggers on both closed (label already present at merge time) and labeled (added afterward — the more common case in practice) against the action's default label_pattern, which already matches this doc's backport <branch> convention with no configuration needed.

Still not possible: no ruleset predicate can require that a commit be a cherry-pick, or verify its referenced original is an ancestor of main — GitHub's rule catalogue has no provenance/ancestry predicate, which is exactly why the check above has to live in CI rather than a ruleset. A full "reject non-cherry-pick commits" gate is not something any project was found running; real projects (Kubernetes, Microsoft) enforce this direction via a restricted list of who can push to the branch plus human review, not commit forensics. The ReleaseBranchProtection/ServiceTagProtection rulesets are exactly that restricted-push mechanism; treat cherry-pick-only as reviewed convention backed by them plus the ancestry check, not a fully automated content gate.

This does not change the "no environment branches" decision above

Branching strategy settled a different axis — no branch governs promotion between dev/preprod/production tiers; the manifest alone does that. A retroactive release branch is orthogonal: it exists only to isolate patches to an already-shipped version from unrelated main churn, has nothing to do with environments, and most versions will never need one at all — only a version that's actually needed a patch after main moved on requires a branch, and it's deleted once that's no longer live anywhere.

A road not taken, worth knowing about

Not every company at scale does this. Uber runs its services from one monorepo with no release branches or independent per-service versioning at all — every production release builds from main continuously, with tiered rollout and automatic halt-on-alert doing the safety work that per-service tags and patch branches do here. Meta draws the same line differently for two products in the same company: web pushes from main every few hours (frequent enough that a two-week-old deployed gap rarely opens), specifically to avoid needing hotfix branches at all, while mobile keeps the branch/cherry-pick model because app-store review cycles force exactly the "already shipped, trunk has moved on" gap this section addresses. That's the same forcing function this repo has: infrequent, manual deploys mean the gap between "deployed" and "trunk's tip" opens routinely, not rarely — so a retroactive branch, not higher deploy frequency, is the applicable answer here.

Why this costs nothing extra in this repo specifically. Every backend build workflow (build-push-api.yml and its 11 siblings) triggers only on a matching push: tags: ref — no push: branches: trigger exists. A release/1.4.x branch can sit in the repo, accumulating cherry-picks, without triggering a single CI run — nothing fires until a commit on it is actually tagged, and only the tagged service rebuilds. The path-filtering machinery other monorepos need to keep a release branch's CI cheap (computing which service changed, only building that one) is unnecessary here, because the tag — not the branch — is what build workflows react to, and a tag names one service.

Tag naming — do not encode the branch name in the tag. Kept as <service>-vX.Y.Z (api-v1.4.2), not api-v1.4.2-release-1.4.x or similar. Checked against seven real projects' actual tag/branch pairs (Kubernetes, Node.js, Rails, Envoy, Grafana, Kafka) — none encode the branch in the tag; the branch is where a commit was reached from, the tag names the artifact, and the two are only loosely coupled (a commit can sit on more than one branch; the branch may be deleted long after the tag still matters). Branch membership is recoverable when needed via git branch --contains <tag> while the branch still exists, or via the manifest's own record of which release a patch belongs to.

Alternatives considered

Four approaches were evaluated before landing on the retroactive repo-wide branch above. Each is a legitimate pattern in production use elsewhere — the choice below is about fit for this repo's shape (shared tree, shared lockfile, heavy cross-service library coupling, infrequent manual deploys), not a claim that the others are wrong in general.

Approach	How it works	Pros	Cons	Why not chosen here
A — Cherry-pick straight onto main's tip, tag from there	Fix merges to main; tag api-v1.4.2 at main's current commit.	Simplest possible mechanism — no branch at all, one tag, done.	The tag no longer reproduces "v1.4.1 plus just this fix" — it also contains every other unrelated merge since v1.4.1, to api and to shared libraries. Not what the fix was tested against; the exact scenario that motivated this whole section.	Fails the stated requirement outright once main has moved on — this is the "naive answer" the section opens by ruling out.
B — Per-service release branch	Branch release/api-v1.4.x from the tag, scoped to just that service's directory.	Conceptually clean: isolates exactly the one service's patch history.	Doesn't exist as a buildable concept here — every image is built via COPY . . from the entire tree with one shared bun.lock (build/Dockerfile.service:36), so "only api's code at this commit" has no meaning; a per-service branch would still need the rest of the tree checked out to build.	Ruled out by this repo's build architecture, not by preference — see the correction in the warning above (v0.8 → v0.9).
C — Repo-wide branch, repo-wide version (Google's, Kubernetes', Envoy's actual model)	One branch, one version for the whole branch — every service tagged from it shares the same version number, changed or not.	Matches how every large, sourced example (Google SRE book, Kubernetes, Envoy) actually runs this; version number always describes the literal tree state, so the "tag a service that wasn't touched" sharp edge (below) can't happen — there's only one tag.	Costs a rebuild-and-redeploy of all 14 services to ship a fix to 2, and a version number that moves even when a service's code didn't change — works against this repo's already-decided independent per-service semver (Three concepts, kept separate).	Rejected specifically to preserve independent per-service versioning, already a settled decision elsewhere in this doc — not because the pattern itself is unsound. Revisit if the per-service-tag discipline below (only tag what was actually cherry-picked) proves hard to hold in practice; Option C is the fallback with the fewest moving parts.
D — Feature flags instead of a branch	Ship the fix behind a flag on main, flip it off/on without a version bump.	No branch, no cherry-pick, no rebuild — a flag flip is instant and needs no CI.	Only covers bugs in code that's actually flag-gated; cannot help with a crash, a bad migration, or wrong logic in an always-on path — which is most of what "a critical bug in production" turns out to be. No flagging framework exists in this codebase today (a separate, real gap).	Not a substitute for the branch — a complementary tool for the subset of fixes it can reach. Worth adopting for its own reasons (faster behavior changes generally), not as this section's answer.
E — Roll back, fix forward at leisure	Point the manifest at the previous good version instead of patching forward.	Fastest possible mitigation — no fix needed at all to stop the bleeding; the manifest already makes this a first-class, equally-fast operation (see Rollback design).	Only available if nothing about the bug requires the new behavior to stay live (a regression, not a needed fix), and breaks down across a backward-incompatible DB migration — rolling back code doesn't roll back schema (see Database migrations).	Not an alternative to the branch, but the first thing to reach for before it — if rollback resolves the incident, there's no need to patch forward under pressure at all. The branch is for the residual case: the fix itself must go out because rollback isn't viable or isn't sufficient.

Decided: retroactive, repo-wide, ephemeral branch (Option B corrected → this)

Chosen because it's the only option that (a) actually builds in this repo's shared-tree architecture, (b) preserves the already-decided independent per-service versioning, and (c) matches documented practice at the trunk-based companies whose model this repo is otherwise following (Google, and Meta for the services where deploy frequency alone can't close the gap). Options D and E aren't rejected — they're the correct first response to try, with the branch reserved for whatever neither can resolve.

Temporal workers — the asymmetric case

4 of the 12 blitz services are Temporal workers (wfw, wfwpdf, classicml, bapiproxy), and wfw alone hosts 7 separate Worker.create() instances in one process across 7 task queues. Rolling back a worker's image is not the same as rolling back its effect — workflow code must be deterministic, and a new binary deployed mid-flight can break replay for in-progress executions.

Worker Versioning exists, is GA, and solves this properly. Temporal's Worker Deployments / Worker Versioning API went GA in March 2026. It lets multiple worker versions run concurrently against the same task queue, ramps a percentage of new workflow starts to a new version, and lets each execution pin to the version it started on (PINNED) or ride forward (AUTO_UPGRADE). Rollback of routing is instant for new workflow starts — but genuinely asymmetric: an execution already pinned to a broken new version doesn't automatically move back; it has to be fixed forward.

Recommendation — phase this in later, not now

Full Worker Versioning requires wiring workerDeploymentOptions into every Worker.create() call, choosing pinned-vs-auto-upgrade per workflow type, and building your own ramp/drain orchestration on Fargate (the Kubernetes-only Worker Controller doesn't apply here, and it's still Public Preview even where it does). That's real machinery for a first release-management pass. The lighter, sufficient first step: disciplined use of the existing patching API (patched() / deprecatePatch()) for any workflow-code change, plus replaying new worker builds against recent production Event Histories before deploying (Temporal's own documented safe-deployment pattern). Revisit full Worker Versioning once there's a concrete incident, or once a long-running workflow class (multi-day close/reconciliation is the obvious candidate here) makes an in-flight determinism break a realistic risk rather than a theoretical one.

Practically, since all 7 of wfw's workers ship in one container image and redeploy together regardless, the natural unit of versioning — if and when this is adopted — is one Worker Deployment Version applied identically across all 7, not independent per-queue versions.

Database migrations — the actual blocker

This is more urgent than image versioning

No service applies Prisma migrations anywhere — not at container start, not in CI, not as a deploy step. Confirmed three ways: zero matches for migrat|prisma across every CI workflow; every Dockerfile's release stage copies only compiled output and node_modules, no prisma/migrations/ tree, no Prisma CLI; and of nine data packages, only one (systemd) even declares a prisma:deploy script — the seven finance-schema packages don't. New tenant databases are created by CREATE DATABASE ... TEMPLATE, not migration, so tenant_base itself has no version identity.

The consequence for this whole design: rolling back a service's image does nothing to roll back its schema. Old code lands on new schema with no compensating action, because there's no forward migration ledger to compensate against in the first place — and no down-migrations exist anywhere. The only rollback mechanism that currently exists is a full destructive restore, hard-gated to staging.

This doesn't block shipping the manifest/tagging/deploy-path work above — those are safe and valuable independent of migrations. It does mean the rollback story is incomplete for any release that includes a schema change until migrations are baselined with a real ledger (_prisma_migrations) and a backward-compatible expand/contract discipline is adopted. Recommend treating this as a parallel, not sequential, workstream — it's gating for AWS production go-live specifically, less so for the Railway/dev identifiability goal.

Phased rollout

Phase 1 — Activate tagging, stop the invisible skip. Add tags: triggers to the build workflows (the derivation logic already exists and is dead code). Add health routes to the 11 services missing one, and fix blitz-api's path mismatch. Fix the two live defects in release-build.ts/release-staging.yml before anyone relies on them.

Phase 2 — Manifest + Railway image converge + GHCR retention. Introduce the manifest at railway/environments/staging/release.json, per the schema above (minimal scope — type + seed file + the image-converge step in provision.ts/deploy.ts — done; see railway/shared/manifest.ts). Still outstanding: the digest-join drift check against Railway's Deployment.meta.imageDigest, and the GHCR retention job — best built once a real corpus of tagged versions exists, since the manifest is what tells it what's safe to delete.

Phase 2.5 — One source of truth for image version. Done; see Phase 2.5 under Railway deploy path. infrastructure.ts's image is now bare/tagless (infrastructure only — ports, resources, env, topology); the manifest is the sole source for version, converged unconditionally (no live "is this a no-op" comparison against Railway or a stale local baseline). provision.ts's brand-new-service bootstrap is the only place a tag (:latest) still gets set directly, and only once, at create time.

Phase 3 — Health-gated rollback on Railway/dev. Not started; scoping only, done 2026-09-06. Deferred in favor of finishing the happy path across all 12 services first — only emailprocessor has ever been through a real, verified manifest-driven tag → build → deploy cycle; rollback is premature before that's true fleet-wide.

Known design gap found while scoping Phase 3: a revert can fail identically to

what it's reverting from "Failed deploy" today (shared/wait-for-deploy.ts) means Railway's own container-level deployment status (FAILED/CRASHED/REMOVED/TIMEOUT) — the same uniform signal for all 12 services regardless of /health route, since 5 of 12 (emailprocessor, bapiproxy, wfw, wfwpdf, classicml) have healthcheckPath: null and this signal was never route-dependent to begin with. That part is fine. The real problem: this signal cannot distinguish "the new version regressed" from "an external dependency this service needs is down" — and a naive auto-revert would make the wrong call in the second case.

Concrete scenario: wfw depends on Temporal at startup. If Temporal itself is down (or scaled down — this happened for real on 2026-09-06, see Current state, verified) and a new wfw version is deployed at that moment, it will crash-loop trying to connect — through no fault of the new version. An auto-revert triggered by that crash would roll wfw back to its previous version, which depends on Temporal identically and would crash-loop the exact same way. Worse, this could read as "the rollback itself failed," when the actual fix has nothing to do with wfw's version at all — it's "bring Temporal back up." A system that can't tell these apart risks either flapping between versions or masking the real incident behind a version-management retry loop.

Resolution, when Phase 3 is actually built: revert must never be a fully automatic action. The design settled on: detect a failed deploy (per-service, container-level, scoped to only the service that failed — a tier deploy's other, successful services keep their new versions untouched) and open a PR proposing the revert, but never merge or redeploy it automatically. A human reviewing the failure's actual logs (in wfw's case, a Temporal-connection error is immediately recognizable as "not this service's fault") is the only reliable way to tell a genuine regression from an environmental outage — this is exactly why the design doc's existing "a human always merges" rule extends to reverts too, with no exception. The previous value to revert to should come from the manifest's own git history (git show HEAD~1:<manifest path>), not a live pre-converge read of Railway's current config, for the same reasons the rest of this design avoids asking the platform what it currently thinks is true.

Phase 4 — Bring AWS preprod to parity. Port blitz-apimgmt and blitz-bapiproxy into aws/modules/ecs/services.tf, and bring aws/environments/production's module set up to match preprod (rds_autostop, elasticache). Mechanical Terraform, but it has to land before Phase 5 can mean anything — preprod can't gate prod on a service or module it doesn't have.

Phase 5 — ECS task definitions with circuit breaker, on preprod. Port the deploy script to aws ecs register-task-definition + update-service, reading the aws-preprod.json manifest. Circuit breaker + alarms on all services; native blue/green on the two ALB-fronted ones. Replace the stability-waiter with DescribeServiceDeployments polling from the start — don't inherit the false-green trap. Full rigor here, since preprod's entire job is being a faithful rehearsal of prod.

Phase 6 — AWS production promotion of record. Same mechanism as Phase 5, pointed at aws-production.json. Railway/dev remains the fast-iteration tier; AWS preprod remains the permanent pre-prod gate, not a one-time migration step. Cross-tier manifest diff job (dev vs. preprod vs. prod) becomes worth having once all three run continuously side by side.

Independent of the phases above, and not gating them: baseline the Prisma migration ledger for the finance schemas. This is its own project with its own risk profile — sequence it in parallel, targeted to land before Phase 6 rather than before Phase 2.

Decisions made

1. Manifest location — inside fin-infra. Co-located with the IaC that reads it, not a separate repo.
2. Branching strategy — Option A. No environment branches; the manifest alone governs what's deployed where. The already-planned staging-branch move stays worth doing for its original, unrelated reason (the hardcoded-main build-source issue), but is not a promotion gate.
3. Approval gate — PR-gated everywhere, no asymmetry between environments. A deploy workflow is the only thing that ever proposes a manifest change — nobody hand-edits the JSON — but it always opens a PR, and a human always merges it, in Railway/dev and AWS preprod exactly the same as AWS production. See The release manifest for the equally important ordering rule this surfaced: the PR is built from what was confirmed successful per service, never from what was planned, and never opened before deploy verification completes.
4. Release naming — stable name, tracked variance. A partial single-service rollback does not mint a new release identifier; it's recorded as a variance from the last whole-set release. See Partial rollback.
5. Frontend hosting — stay on nginx containers for now. blitz-ui and blitz-mgmtui remain Railway/ECS nginx containers on AWS. S3+CloudFront is not ruled out, but it's a separate piece of work needing its own thorough testing (CORS from a split origin, the private-backend proxy hop blitz-ui's nginx currently performs, cache invalidation on deploy) — not something to fold into this release-management effort. Revisit once blitz-api/ws are stably ALB-exposed on AWS.
6. Migration sequencing — ships in parallel, not gating. Baselining the Prisma migration ledger (see Database migrations) is its own workstream, targeted to land before Phase 6, but does not block Phases 1–5.

Still open

None remaining from the original seven — see item 7's resolution below.

7. GHCR retention window — resolved. scripts/cleanup-ghcr.sh (run weekly by cleanup-ghcr.yml) now keeps, per service: the latest tag; whatever tag is currently pinned in the live manifest (railway/environments/staging/release.json), regardless of age — this is the "referenced by a manifest still considered live" criterion from GHCR growth & retention above, not a bare count; and the last N=5 most-recent builds by version ID as a short buffer for anything not yet promoted through the manifest. A semver tag that isn't manifest-pinned gets no special protection beyond that N=5 window — it ages out like any other build, closing the earlier "keep every semver tag forever" gap. N=5 was chosen to match the pre-existing recency buffer rather than a fresh judgment call; revisit if a real rollback ever needs to reach further back than that.

Changelog

Version	Change
v0.1	First-cut draft — per-service semver tags, single aggregate manifest, ECS circuit breaker + Railway health-poll rollback. Written before the codebase was directly audited.
v0.2	Rewritten against a verified audit of all three repos plus live Railway/GitHub API checks. Corrected the manifest/tagging premises, flagged the migration blocker, scoped Temporal Worker Versioning as phase-2.
v0.3	Added the three-tier environment model after Railway was clarified as dev-shaped, not a true pre-prod; added the branching strategy, GHCR retention, and partial single-service rollback sections, all previously unaddressed; clarified blue/green as fully native and ECR-vs-GHCR immutability mechanics; corrected the AWS ECS module's missing-services gap with dates; fixed a confusing service-count table; noted the missing-/health gap has no tracked issue; renumbered phases around the three-tier promotion chain.
v0.4	Resolved 6 of 7 open decisions: manifest lives in fin-infra; no environment branches; every environment's manifest change goes through a PR a human merges, no asymmetry between dev and production; partial rollbacks keep the release name stable and track a variance instead of minting a new one; frontend hosting stays on nginx containers, S3+CloudFront deferred pending its own testing; migration-ledger baselining ships in parallel, not gating. Corrected a real ordering bug this surfaced: the manifest must be written from what deploy verification confirmed succeeded per service, never from a pre-planned diff or a bare job-exit-code check — added as an explicit rule under The release manifest, and the "commit that, deploy it" ordering in Partial rollback was fixed to match. GHCR retention window (item 7) remains the one open decision.
v0.4	Converted from the working artifact into this fin-internal-docs page; no content changes beyond format (plain Markdown/VitePress in place of custom HTML).
v0.5	Closed a gap surfaced by "how do I decide when to tag if main keeps moving": deploys are now tag-first on every tier, including dev — nothing deploys :latest, not even for fast iteration. Added the concrete tag-then-deploy sequence under Tagging scheme (tag an exact commit before building, deploy that tag, promote the same already-built artifact by reference — never derive "what's actually running" after the fact by querying Railway/ECS). Corrected the three-tier table and manifest digest explanation, which previously implied dev could stay on a looser :latest-ish artifact; dev's actual speed advantage is a fast tag-and-review cycle, not a looser artifact or a skipped PR review.
v0.6	Renamed the manifest's live environment from railway-dev to railway-staging throughout, and its file path from release-manifests/railway-dev.json to railway/environments/staging/release.json — both to match this repo's pre-existing convention (railway/environments/staging/variables.env.example, RAILWAY_ENVIRONMENT default 'stage') instead of introducing a second name for the same live environment. Surfaced by implementing Phase 2 (minimal scope): the manifest type, seed file, and image-converge step in provision.ts/deploy.ts landed in fin-infra, and its dev/ directory collided with the existing staging/ one. Also recorded emailprocessor's real, live-tested tag (emailprocessor-v0.1.0) as the manifest's first genuine non-placeholder entry — the first end-to-end proof the tag-first build trigger actually works.
v0.7	Added the Current process vs. proposed process diagram summarizing the current release chain, its five gaps, the proposed tag-first/manifest-driven process, and how each gap closes — placed right after the intro so a reader gets the shape of the design before the detailed prose. Updated the top-of-page Status line to reflect actual implementation state (Phase 1 live, Phase 2 in code review) rather than a blanket "not yet implemented."
v0.8	Closed a real gap surfaced by "how do I patch one service after main has moved on": added Patching one service without dragging in the rest of main — a per-service release branch, cut reactively (not pre-emptively) from the exact patched-from tag, fixed forward-on-main-first and cherry-picked backward, with a label-triggered backport bot recommended to keep the cherry-pick mechanical rather than manual toil. Grounded in industry research (Kubernetes, Node.js, Rails, GitLab Flow, Google SRE, Trunk-Based Development all converge on the same fix-forward/cherry-pick-backward direction) with an explicit counter-example (Uber's monorepo, which uses no per-service branches or versioning at all) included for balance. Clarified in Branching strategy that this is orthogonal to, not a reversal of, the "no environment branches" decision.
v0.9	Corrected v0.8's framing after deeper research: "trunk-based ⇒ no release branches" was the wrong premise — Google's own SRE book documents branching from the mainline and cherry-picking fixes into it as its standard release process, and trunkbaseddevelopment.com names this "branch for release" / "retroactive branching," the sanctioned trunk-based mechanism for patching an already-shipped version. Rewrote Patching one service without dragging in the rest of main: the branch is now repo-wide (this repo's shared tree/lockfile/heavy cross-service library coupling — top shared libs used by 9 of 11 deployed services — makes a true per-service branch incoherent, since every image is built via COPY . . from the whole tree), ephemeral (cut just-in-time, deleted once the patch is no longer live anywhere), while tags stay per-service — with an explicit new warning that only the service(s) actually cherry-picked onto the branch should be tagged from it, since a repo-wide branch can otherwise silently let one service's tag carry another's unrelated cherry-pick. Added a warning on what's actually enforceable in CI today (checked directly against this org's GitHub rulesets: no tag ruleset exists in blitz/fin-infra, the one ruleset present is org-level and scoped to the default branch only, and no GitHub predicate can verify a commit is a genuine cherry-pick or trace its ancestry — git merge-base --is-ancestor is the closest real check, enforcement in practice is a restricted push list plus human review, not commit forensics). Corrected the tag-naming guidance to cite verified real examples (Kubernetes, Node.js, Rails, Envoy, Grafana, Kafka — none encode branch name in tag) rather than asserting an "anti-pattern" no source actually calls it. Replaced the Uber counter-example's framing with the sharper, sourced distinction: Meta runs the same trunk-based company two ways for two products — frequent-enough web pushes avoid needing hotfix branches at all, while mobile keeps branch/cherry-pick because app-store review cycles force the same "already shipped, trunk moved on" gap this repo's infrequent manual deploys also force. Extended the Current process vs. proposed process diagram with a third section showing the retroactive-branch patch flow end to end (deployed tag → fix merges to main → branch cut from that tag → cherry-pick → tag only the patched service → branch deleted once superseded).
v0.10	Added explicit "alternatives considered" pros/cons tables to the three decisions in this doc that previously stated only the chosen approach in prose: tag-first vs tag-after-deploy vs dev-only-latest; the four approaches evaluated for patching one service (straight-cherry-pick-onto-main, per-service branch, repo-wide-branch-repo-wide-version, feature flags, and rollback-first, each with why it wasn't chosen here specifically); and stable release name vs minting a new one per change vs no identifier at all. Matches the pros/cons table format already used for Branching strategy (Option A/B) and the AWS deployment strategy comparison elsewhere in this doc, rather than leaving these three decisions justified only by prose.
v0.11	Added Phase 2.5 — one source of truth for image version, surfaced by auditing the as-shipped Phase 2 code against this doc: infrastructure.ts's hardcoded :latest and the manifest were left as two independent, unreconciled sources for image version, with the manifest consulted only when deploy.ts --from-manifest is passed explicitly — not a bug in either file, but an unfinished migration with no tracked task before now. Scoped the fix and flagged it as blocked on every service carrying a real manifest entry first.
v0.12	Corrected v0.11's sequencing premise and shipped Phase 2.5: the planned fix would have compared the manifest against a "currently declared" baseline (infrastructure.ts, then — once that stopped carrying a real tag — Railway's live service-instance state) before converging, which is exactly the "ask the platform what should be true" pattern this design exists to avoid; the version is decided once, at tag time, and should only ever need to be read from the manifest. Fixed by making convergeServiceImage converge unconditionally whenever a manifest entry exists — no comparison, no baseline — since updateServiceInstance is idempotent and a same-value call is a harmless no-op. This also removed the blocking dependency v0.11 assumed: no need to wait for every service to carry a real manifest entry, since there was never a comparison to trip over. Landed: manifest.ts's convergeServiceImage drops its declaredImage parameter; apply-manifest-images.ts/deploy.ts drop their infrastructure.ts-derived baseline maps; infrastructure.ts (backend + frontend) makes image bare/tagless; provision.ts gets a bootstrapImage() helper for the brand-new-service create-time case (the only place a tag is still set directly); check-image-drift.ts needed no change (it only ever used .image for the bare name, not the tag). Updated the top-of-page Status line, Phased rollout, and removed the now-resolved item 8 from Still open.
v0.13	Began implementing Patching one service without dragging in the rest of main's retroactive-branch tooling. Shipped: ReleaseBranchProtection and ServiceTagProtection repo-level rulesets on both blitz and fin-infra (require PR + 1 approval to merge into release/**, block force-push/deletion there; block deletion/force-update of any tag while leaving creation open) — closing the two concrete gaps the "What's enforceable in CI" callout had flagged. Also added two corrections surfaced by direct questioning of the retroactive-branch design: an exception to "fix lands on main first" for the case where the bug doesn't reproduce on current main at all (already fixed by coincidence, materially different after conflict resolution, or the code is gone entirely — the last case genuinely has no main-side fix to write, with an admin/moderator bypass as the escape hatch); and a new known-gap warning for patching an already-patched release line (reuse the branch if it still exists; if deleted, re-cut from the latest surviving tag, not the original — and nothing today automates the "provably dead everywhere" check the manifest could answer before a branch is deleted). Backport automation (korthout/backport-action) and the merge-base --is-ancestor CI check remain not yet built.
v0.14	Finished implementing Patching one service without dragging in the rest of main's retroactive-branch tooling: the two items v0.13 left open. cherry-pick-ancestry-check.yml (reusable, in fin-infra, called cross-repo from blitz matching the sync-finboard.yml/lockfile-guard.yml pattern) runs on every push to release/**, extracting each new commit's (cherry picked from commit ...) trailer (if present) and rejecting the push if the claimed origin isn't actually an ancestor of main — verified locally against a genuine cherry-pick (passes), a fabricated trailer (fails), and a plain commit with no trailer (silently skipped, since this cannot enforce "must be a cherry-pick," only "if it claims to be one, prove it"). backport.yml (both repos) wires up korthout/backport-action@v4.6.0 on the maintainer's own documented pull_request_target trigger, extended to fire on both closed and labeled (the README's example only covers a label already present at merge time; this repo's actual usage — labeling an already-merged PR after the fact — needed the second trigger type added). All four items from Patching one service's original scope are now shipped.
v0.15	Resolved item 7 (GHCR retention window), the last of the original seven open decisions. Audited scripts/cleanup-ghcr.sh (fin-infra#108) against this doc's own GHCR growth & retention criterion — "retain every version referenced by any manifest still considered live" — and found the shipped script didn't implement that: it kept latest + any semver tag forever + last 5, meaning semver-tagged versions accumulated without bound as more services get tagged. Fixed by reading the live manifest (railway/environments/staging/release.json) directly in the cleanup script: the version pinned there for a service (once set — still "unset" for all 14 services as of this revision) is now the only tag exempt from the recency window, matching the doc's actual criterion instead of a blanket semver carve-out. Last-5-by-recency (KEEP_RECENT=5, unchanged) remains the buffer for anything not yet promoted through the manifest. Verified against live GHCR data for blitz-emailprocessor (13 real versions) before merging: correctly kept latest + 4 more recent builds, correctly dropped the now-unpinned emailprocessor-v0.1.0 test tag out of the window. Also, separately: Phase 4 (AWS preprod parity — blitz-apimgmt/blitz-bapiproxy added to aws/modules/ecs/services.tf, production root brought to preprod's module set) shipped as fin-infra#115. AWS Phase 5/6 work is intentionally deferred to its own thread, not part of this pass.
v0.16	First real end-to-end test of the manifest-driven Railway deploy path: tagged emailprocessor-v0.1.1 on blitz main's tip, watched the build through GitHub Actions, then independently verified the pushed GHCR image via docker inspect/buildx imagetools inspect (ground truth, not workflow-log trust) — confirmed org.opencontainers.image.revision matched the exact tagged commit and org.opencontainers.image.version read 0.1.1. Surfaced and documented a genuine, previously-unknown gap along the way — see the new Current state, verified callout: 4 of 12 backend services (emailprocessor, wfw, wfwpdf, classicml, all pure workers with no HTTP surface) have no version.ts//api/version route at all, so there is no way to confirm from a running container which version is actually executing — distinct from, and previously conflated with, the already-documented /health coverage gap. Recorded as a known limitation rather than fixed inline, since it doesn't block this test and deserves its own scoped fix.
v0.17	Corrected v0.16's premature "fully proven end to end" claim, and fixed the real bug it was hiding: continuing the live E2E test by scaling up dependencies (Temporal/Valkey were also scaled down, not just wfw) and confirming emailprocessor genuinely running (real IMAP polling, not just a passing health check) — then a Railway dashboard screenshot showed the ACTIVE deployment still on :latest, contradicting the manifest's emailprocessor-v0.1.1. Root-caused to a real, pre-existing bug: serviceInstanceRedeploy restarts the previous build and ignores a source.image update made just before it, confirmed on 6 consecutive real deployments back to 2026-08-30. Fixed by introducing deployService (serviceInstanceDeploy, which does resolve source fresh) for deploy.ts's manifest-converge path specifically — see the new Critical bug found and fixed callout under Railway deploy path for the full writeup. Also added, per explicit request: a version-numbering convention (operational semver: MAJOR = requires action beyond deploy, MINOR = new capability, PATCH = no contract change; independent per-service, starting at 0.1.0) and its CI enforcement, Version monotonicity check — a new reusable version-monotonicity-check.yml (fin-infra, called from blitz/blitz-ui) that rejects a new <service>-vX.Y.Z tag whose version isn't strictly higher than every prior tag for that service, explicitly scoped to never block rollback (redeploying an old tag creates no new git tag, so the check never runs for it). Confirmed deliberately NOT adopting semantic-release/changesets — built for package registries, not container images, and semantic-release's Conventional-Commits prerequisite is a poor proxy for the operational blast-radius judgment the bump rule actually needs.
v0.18	Resolved the runtime version-visibility gap v0.16 flagged, and corrected its scoping in the process. Rechecking excelrw (challenged directly — "bapi and excelrw are not workers, it should have had endpoints") found the earlier claim wrong: excelrw already has GET /version (VersionController.cs), missed the first time by grepping the wrong string. The real gap was narrower there but wider elsewhere: bapiproxy (.NET, genuinely HTTP-serving) does NOT share excelrw's controller — separate .csproj, no project reference — so it had the same blind spot as the Bun workers despite having an HTTP surface; and classicml (Python — a third runtime exception alongside the two .NET services, confirmed via fin-infra's CLAUDE.md service table and its scikit-learn/FLAML dependencies, which have no TypeScript equivalent) turned out to have never had the SERVICE_VERSION bake wired into its Dockerfile at all, a deeper gap than "just missing an endpoint." Fixed all of it in blitz#1439: emailprocessor/wfw/wfwpdf log version.json in their startup line; bapiproxy gets its own VersionController.cs mirroring excelrw's, exposing GET /bapi/version; classicml gets the bake added to Dockerfile.py.service plus a runtime read in main.py. Not compile/build-verified for the .NET controller (no local dotnet SDK) or live-tested (no tagged build run yet) — flagged honestly in the PR rather than claimed as proven.
v0.19	Designed and shipped tag-time manifest automation + manifest-diff-driven deploy, end to end, in two pieces (full plan reviewed and approved before implementation, given the scope). Piece 1 (update-manifest-from-tag.yml in fin-infra, dispatch-manifest-update.yml in blitz/blitz-ui): after a service tag's build succeeds, a workflow_run-triggered dispatch (not a second tags: trigger + polling loop — fires exactly once per real build completion) calls fin-infra's Actions API to run a same-repo workflow that independently re-verifies the image's digest in GHCR (never trusts a value passed across the dispatch payload), edits exactly that one service's manifest entry via a new update-manifest-entry.ts, and opens a PR — never auto-merges, a human still reviews and merges every manifest change exactly as today. Each run uses its own fresh branch (manifest/<service>-<version>-<run-id>), so two different services' tags landing seconds apart get independent, non-conflicting PRs. Requires a new secret, FIN_INFRA_DISPATCH_TOKEN (actions: write on fin-infra only), held in blitz/blitz-ui — provisioning tracked separately, not done by this automation itself. Caught and fixed a real mapping bug before shipping: the build workflow's tag prefix (api, apimgmt, ui, mgmtui) differs from the manifest's canonical service key (blitz-api, blitz-apimgmt, blitz-ui, blitz-mgmtui) for exactly these four services — getting this wrong would have silently created a duplicate, wrong key instead of updating the real one. Piece 2 (deploy-from-manifest-diff.yml + deploy-from-manifest-diff.ts in fin-infra): a workflow_dispatch-only workflow (deliberately not triggered by the manifest PR merging — keeps "approve the content" and "deploy now" as two separately-controllable human actions) that diffs the manifest's current committed content against its immediately previous commit (structural field comparison, not text diff; git history only, never a live Railway read) to find which service(s) actually changed, independently re-verifies each changed service's image still exists in GHCR at the claimed digest (fails loudly on mismatch rather than deploying whatever a tag currently resolves to), then redeploys only those services. Required a new deploy.ts capability (a comma-separated multi-service target, e.g. recon,wfw, filtering STARTUP_TIERS to preserve real relative ordering across the subset — verified against real tier boundaries and an unknown-service-name case, which now fails loudly instead of silently skipping). Doubles as rollback with no separate code path: a human reverting a manifest entry in a normal PR is, to this workflow, indistinguishable from any other "changed" case — verified by reverting a throwaway test entry and confirming the diff-and-redeploy logic picked it up correctly. Added a doc clarification distinguishing this automation's narrower claim ("this image was built and verified in GHCR") from the existing, stronger "manifest asserts a confirmed deploy" rule — a merged Piece-1 PR is not proof staging is running that version until Piece 2 has actually run.
v0.20	First real fleet-wide proof: all 14 services tagged, built, verified, and deployed in one session using the v0.19 automation, all independently confirmed via Deployment.meta.imageDigest. Also added a new Tag Release workflow (tag-release.ts) — computes the next version from the manifest's current entry + a chosen bump (patch/minor/major), previews it, and only pushes the real tag on confirm=true; a belt-and-suspenders guard compares the candidate against the highest REAL existing tag for that service (not just an exact-name check) after an early version of this guard let a stale candidate (recon-v0.1.1, computed from a manifest not yet caught up to a merged-but-unmerged recon-v0.1.2 tag) get pushed and only then get rejected by version-monotonicity-check.yml — fixed same night. New secret RELEASE_TAG_TOKEN (Contents: write on blitz + blitz-ui only). Found and documented a real gap in deploy-from-manifest-diff.ts's diff window: it always compares HEAD~1 vs HEAD, so batching two manifest PRs' merges before running the deploy workflow once causes the earlier PR's change to be silently invisible — hit for real when agents (PR #130) was skipped after PR #143 (12 other services) merged next and the deploy workflow only ran after both. Fixed the immediate case by hand (direct deploy-staging.yml --from-manifest agents dispatch); a code-level fix (a persisted lastDeployedSha on the manifest) was scoped but not built, since main's branch protection requires a PR for every commit and there is no way to grant that one automated write bypass access without granting the same actor blanket unreviewed push access to main — see the new operational-rule callout above. Also: disabled Notify Push (main) in all three repos (fin-infra, blitz, blitz-ui) to cut Actions-minute spend from the team's routine daily merge volume, separate from any release-batch cost — re-enable is a manual gh api .../actions/workflows/<id>/enable per repo, tracked in memory rather than this doc since it's an operational toggle, not a design decision.

References

• Working design artifact (Claude-generated, iterative source of this page)
• fin-infra CLAUDE.md — AWS Production, Resource limits, Secrets sections
• fin-infra#35 — no container image scanning, signing, or digest pinning; :latest tags in prod
• Deploy Scope & Digests — the prior decision note this page supersedes the recommendation of ("pin :sha tags... tie this to how we want rollbacks to work")
• Environments — current environment inventory this page's three-tier model extends
• Rollback — will need updating once this design is implemented; not yet touched by this proposal