AWS: preprod choices and what production must change

Status: live reference · Last updated: 14 September 2026

Preprod was brought up on 14 September 2026. Several settings were chosen deliberately for a low-cost POC tier and are wrong for production. This page records each one, the reasoning, and what production needs instead — so the production root module is not created by copying preprod.

The companion pages are AWS architecture (the target design) and AWS implementation log (the phased build record). This page is only the preprod/production divergence.

Read before creating the production root module

aws/environments/production/main.tf already exists and is at module parity with preprod. Every row in the table below is a place where copying preprod's values would be a defect.

The divergence table

Area	Preprod today	Production must be	Why
Capacity provider	FARGATE_SPOT	FARGATE	Spot tasks are reclaimed with 2 minutes' notice. Acceptable for a POC, an outage in production. Already pinned in production/main.tf.
Task CPU/memory	0.25 vCPU / 512 MB, all 12	Measured per service; see sizing	Preprod starts at the Fargate floor and raises on evidence. Production must not inherit values chosen for cost.
desired_count	1 (workers 0)	≥ 2 for blitz-api, blitz-ws	One task means a Spot reclaim or a rolling deploy is a gap in service.
RDS	db.t3.small, single-AZ, no deletion protection	Multi-AZ, deletion protection ON	Preprod holds restored dumps; production holds the only copy.
skip_final_snapshot	true	false (the module default)	Preprod databases are restored from dumps and hold nothing durable.
restore_enabled	true	false — never true	Enables apimgmt's destructive drop+restore endpoint. Already false in production/main.tf.
Bastion	bastion_enabled = true	false, raise only for a task	An always-on SSH-less jump host is still an always-on entry point.
Interface VPC endpoints	none	reconsider at production traffic; logs first	See VPC endpoints.
Log retention	30 days	longer, per the retention policy	Compliance, not cost.
Temporal	not connected	Temporal Cloud, mTLS	Workers sit at desired_count = 0 until then.

Cost decisions and the reasoning

Fargate Spot

Preprod runs every task on FARGATE_SPOT — roughly 70% cheaper. All 12 services at the floor cost about $25/month running versus ~$83 on-demand.

What a reclaim actually does: AWS sends SIGTERM, deregisters the task from the ALB target group, waits 120 seconds, then SIGKILL. ECS launches a replacement immediately (deployment_maximum_percent = 200), so the sequence is notice → replacement starts → old task drains → old task dies.

At desired_count = 1 there is still a gap of roughly one task-startup (30–90s) per reclaim. Temporal workers absorb this naturally — the server holds workflow state and activities are retried elsewhere — so a reclaimed wfw is a non-event. WebSocket connections on blitz-ws are the real casualty.

Graceful shutdown is not yet verified

ECS task definitions do not set stopTimeout, so containers get the ECS default of 30 seconds, not the 120 that Spot allows. Whether the Bun services actually drain on SIGTERM is unverified. This matters for ordinary rolling deploys too, not just Spot.

Task sizing

Every service is at 256 CPU units (0.25 vCPU) / 512 MB — the cheapest task Fargate sells, and the only tier that permits 512 MB at all.

Two facts that should drive every future sizing change:

Fargate accepts only fixed cpu/memory pairings. 0.5 vCPU requires ≥ 1 GB; 1 vCPU requires ≥ 2 GB; 2 vCPU requires ≥ 4 GB. An invalid pair is rejected by RegisterTaskDefinition at apply time, not at plan time — a check block in modules/ecs/services.tf now surfaces it during plan.

vCPU costs about 9× what memory does per unit — $0.04656/vCPU-hr against $0.00511/GB-hr in ap-south-1. So +0.25 vCPU is +$8.50/month while +1 GB is +$3.73. When a service needs headroom, raise memory before CPU.

Two services have known OOMs at these values

Railway raised both after confirmed crashes, and preprod is deliberately running below those values:

• blitz-api — Railway runs 1 GB; 512 MB OOM-confirmed. Expect this to be the first to need raising. 256/1024 costs only +$1.87/month.
• blitz-wfw — Railway runs 1.5 GB; OOMs below 1 GB with 5 parallel Temporal workers. Currently desired_count = 0, so not yet exposed.

The tell is a container that dies with a clean log — that is SIGKILL, not a crash.

Preprod values intentionally do not match railway/shared/resources.ts. Railway's are the measured ones; matching them would roughly double preprod's compute cost with no load to justify it.

Registry: ECR, not GHCR

Preprod pulls images from ECR, reversing the earlier "GHCR, no ECR migration" decision — a bullet recorded with no rationale, made before AWS was live and therefore before NAT egress existed as a cost.

Two reasons:

Cost. Fargate runs in private subnets, so a GHCR pull crosses the NAT gateway as billable processing plus transfer-out. Measured image sizes average ~264 MB (wfw 361, api 288, apimgmt 278, ws 266, bapiproxy 126), so one full 12-service pull cycle is ~3.2 GB ≈ $0.49. Pulls are frequent: Spot reclaims, restarts, every rolling deploy, scale-up from zero, and QA iteration.

No credential to expire. GHCR needs a PAT in Secrets Manager, rotated by hand. An expired one took the entire preprod cluster down on 14 September 2026 — every service CannotPullContainerError: 403, with no task ever starting. ECR authenticates via the ECS task execution role: nothing to rotate or leak.

GHCR remains the build output and source of truth; images are synced with aws/scripts/sync-ghcr-to-ecr.sh. Railway continues to pull from GHCR unchanged.

VPC endpoints: the free one is the one that matters

ECR is a real service — auth, tags, metadata, scanning, lifecycle — but it stores image layers in S3 and redirects pulls there by presigned URL. So one docker pull touches three paths:

Step	Endpoint	Bytes
auth token	ecr.api (interface, ~$7.30/mo)	~KB
manifest	ecr.dkr (interface, ~$7.30/mo)	~KB
layers	s3 (gateway, free)	~264 MB

Preprod enables only the S3 gateway endpoint. An interface endpoint is a fixed ~$7.30/month bet on volume, and it needs about 47 GB/month of traffic to break even. Nothing at preprod scale comes close:

Endpoint	Est. traffic	NAT cost	Verdict
logs	~1.2 GB/mo	$0.18	NAT cheaper (the closest of the four)
secretsmanager	~1 MB/mo	~$0	fetched only at task start
ssm	~1 MB/mo	~$0	fetched only at task start
ssmmessages	~50 MB/mo	$0.01	only while a bastion session is open

The ecr.api/ecr.dkr pair would need ~165,000 pull cycles a month to justify itself against realistic usage of ~100.

Because there is no fixed endpoint cost, ECR is strictly cheaper than GHCR at every volume — not merely break-even.

Production should re-derive this, not copy it. logs is the candidate: awslogs shipping is continuous rather than per-pull, so it scales with task count in a way the others do not. One caveat to note — with ssmmessages removed, bastion access depends on the NAT gateway being healthy, which is precisely when you might need to get in.

Cost guardrails

A runaway image-pull loop has cost ~$100 once. It recurred on 14 September 2026: eight services retried CannotPullContainerError for ~23 minutes before being scaled to zero by hand.

ECS has no max-retries for task placement. While desired_count > 0 and nothing is running, it retries indefinitely, re-pulling the image each time.

Three defences are in place:

1. Deployment circuit breaker on every service, with rollback. Scope matters: it acts on a deployment in progress, so it catches "a bad image broke a working service" but cannot save a brand-new service whose first deployment never succeeded — there is no previous revision to roll back to.
2. NAT egress alarm — 5 GiB per 5-minute period, minute-resolution. This is the real backstop. The existing AWS Budgets evaluate daily and therefore report the damage after it is done.
3. SNS topic in the preprod account, because CloudWatch alarms can only publish to a topic in the same account and region.

The SNS email subscription needs a confirmation click

Until someone clicks the emailed link the subscription is "pending confirmation" and delivers nothing — indistinguishable from a working alarm path. Verify it after applying.

Operational rule: scale a service to desired_count = 0 as soon as a pull/crash loop is diagnosed, before preparing the fix.

Networking and exposure

• RDS is in private subnets, publicly_accessible = false, and its security group accepts 5432 only from the ECS security group. There is no path from a laptop without the bastion.
• The bastion is a t3.nano in a private subnet with zero ingress rules. SSM works by the agent polling outbound, so there is no open port, no public IP and no SSH key. Access is IAM-gated and every session is recorded in CloudTrail. It is count-gated and should be off when not in use.
• Tenant DNS is a single *.app.finaisse.com CNAME to the ALB with proxied = false. Cloudflare is DNS-only, so AWS WAF on the ALB is the only WAF layer — Cloudflare's does not apply to tenant traffic.
• No Cloudflare Advanced Certificate Manager purchase is required. ACM issues the second-level wildcard free and the ALB terminates TLS. The paid Cloudflare cert would only be needed if that record were flipped to proxied = true.

*.app.finaisse.com is a single shared namespace

Preprod holds the wildcard, so production cannot create the same record while preprod owns it — there is no allow_overwrite, so the second apply fails. Moving it is a deliberate cutover: repoint *.app at the production ALB and every tenant URL follows in one step. Two environments cannot serve .app. tenants simultaneously without per-environment labels.

Still open

Item	Detail
BLOB_HOST / TEMPORAL_HOST	Still Railway hostnames (s3.railway.internal, temporal.railway.internal) and unreachable from AWS.
S3 client is MinIO-shaped	bm.service.ts / fs.service.ts build http://${host}:${port} with forcePathStyle and a hardcoded us-east-1. Real S3 needs no custom endpoint, HTTPS, the correct region, and task-role credentials — a blitz code change, not an SSM edit.
Temporal Cloud	Not purchased. mTLS + namespace endpoint, so host/port alone is insufficient.
Graceful shutdown	stopTimeout unset (30s default vs Spot's 120s); SIGTERM handling unverified.
Identity Center	aws/global/identity-center/ exists but is unfinished. Would replace root/IAM-user/switch-role juggling with one portal login.
Database bootstrap	mgmt cannot be created through the mgmt console — apimgmt hard-blocks its own control-plane DB, and listing restore targets requires a working mgmt. It must be created and migrated over the bastion first.