AWS Production Implementation Log

Tracks actual progress of the Finaisse AWS production setup phase by phase.
Architecture decisions and service mapping: see AWS Architecture.

Started: June 2026
Primary region: ap-south-1 (Mumbai) — preprod + production
Future production region: us-east-1 (planned, not started)
IaC: Terraform (fin-infra/aws/) — all resources via Terraform, no manual console provisioning
Registry: GHCR (unchanged)

Terraform Root Structure

aws/
├── global/
│   ├── bootstrap/         S3 state bucket + DynamoDB lock table (local state, run once)
│   ├── control-tower/     Landing Zone, OUs, SCPs, GuardDuty, budgets, Access Analyzer
│   ├── aft/               Account Factory for Terraform — account vending pipeline
│   ├── identity-center/   Zoho SAML SSO, permission sets, group → account assignments
│   └── iam/               GitHub Actions OIDC provider
├── aft-accounts/
│   ├── preprod/           AFT account request — triggers preprod account vending
│   └── production/        AFT account request — triggers production account vending
├── environments/
│   ├── preprod/           VPC, RDS, S3, KMS, WAF, Secrets, IAM, Flow Logs
│   └── production/        Same — production-grade sizing and settings
└── modules/
    ├── networking/         VPC, subnets, IGW, NAT, security groups
    ├── rds/                PostgreSQL 16, CMK encryption, force_ssl, configurable retention
    ├── s3/                 App data + frontend buckets, SSE, versioning, lifecycle
    ├── secrets/            Secrets Manager — all app secrets, CMK encrypted
    ├── iam/                ECS task execution + task roles
    ├── kms/                CMKs for RDS, S3, Secrets Manager (auto-rotating)
    ├── waf/                WAFv2 — managed rules, SQLi, rate limiting, WAF logs
    ├── flow-logs/          VPC Flow Logs to S3
    ├── elasticache/        ElastiCache Valkey 7.2 (Redis-compatible), at-rest encryption
    ├── ssm/                SSM Parameter Store — non-secret config (80 vars)
    └── ecs/                ECS Fargate cluster, ALB, ACM cert, CloudWatch logs, GHCR pull

Apply order (first-time setup)

1. global/bootstrap         → terraform apply  (S3 + DynamoDB — local state)
2. Console (2 actions only) → Enable IAM Identity Center
                            → Run Control Tower Landing Zone wizard
3. global/control-tower     → terraform apply  (OUs, SCPs, GuardDuty, budgets, alerts)
4. global/aft               → terraform apply  (AFT pipeline infrastructure)
5. aft-accounts/preprod     → terraform apply  (vends finaisse-preprod account)
6. global/iam               → terraform apply  (GitHub Actions OIDC)
7. global/identity-center   → terraform apply  (Zoho SSO, permission sets, groups)
8. environments/preprod     → terraform apply  (all preprod infrastructure)

Phase 0 — Account Foundation

Status: In progress — bootstrap + control-tower + iam done; identity-center remaining
Scope: AWS Organization, Control Tower, IAM Identity Center, account structure

Decisions made

Item	Decision
IaC tool	Terraform (multi-cloud: AWS + Cloudflare + future GCP)
Region	ap-south-1 (Mumbai) — data residency, latency, RBI alignment
Multi-account	Yes — management + preprod + production (separate accounts)
Account governance	AWS Control Tower + Account Factory for Terraform (AFT)
SSO	Zoho SAML → IAM Identity Center (GitHub Team plan excludes SAML)
Control Tower timing	Set up before any member accounts — retrofitting at 5+ accounts is painful (prior experience)
AWS Config	Disabled until SOC 2 prep — re-enable in `control-tower/main.tf`
Hyderabad (ap-south-2)	Rejected — ElastiCache and CloudFront not available
Control Tower vs manual org	Control Tower chosen — guardrails, AFT account vending, audit accounts out of the box

Checklist

One-time manual prerequisites

[x] MFA enabled on root account
[x] IAM user terraform-admin created with AdministratorAccess (631461173580)
[x] Enable IAM Identity Center in console — instance ssoins-65951502a8bf87ba, ap-south-1
[x] Run Control Tower Landing Zone wizard in console — version 4.0, Security OU created

Terraform — global/bootstrap

[x] terraform apply — finaisse-terraform-state S3 + lock via use_lockfile = true

Terraform — provider lock files (2026-09-24)

[x] .terraform.lock.hcl committed in all 7 roots (six aws/ + cloudflare/), hashes for darwin_arm64 + linux_amd64 — fin-infra#297. Before this, they were gitignored and each machine picked its own provider version (preprod and production already disagreed on cloudflare, 4.52.8 vs 4.52.9). This is the provider lock; the S3 state lock above (use_lockfile) is a different mechanism.
[ ] AWS provider 5 → 6 (fin-infra#296, Renovate): run init -upgrade + validate + plan on each root before merging. Merging first makes every plan fail against the old locks.

Terraform — global/control-tower ✅ complete (2026-06-18)

[x] Organization o-1iqncs111v, Infrastructure OU ou-1a46-ykv31eq7 + Production OU ou-1a46-0mkolro2
[x] Log Archive account 430597289467 + Audit account 236353235135 under Security OU
[x] GuardDuty enabled org-wide, delegated to Audit account
[x] AWS Config recorder created but disabled (re-enable for SOC 2)
[x] SCPs applied to Infrastructure + Production OUs:
- Deny non-approved regions (ap-south-1 only)
- Require IMDSv2 on all EC2
- Deny leaving organization
- Deny disabling GuardDuty
- Require S3 encryption
- Deny CloudTrail deletion
[x] Monthly budget alert ($200 org, $100 preprod) + cost anomaly detection ($20 threshold, IMMEDIATE via SNS)
[x] SNS ops alerts topic → sekhar@finaisse.com
[x] IAM Access Analyzer (ACCOUNT scope — upgrade to ORGANIZATION after trusted access setup)
[x] AWS Health events → SNS
[x] Alternate account contacts (billing / security / ops)

Workload accounts (direct, AFT removed)

[x] finaisse-preprod account 219366469126 — under NonProd OU
[x] finaisse-production account 049211686059 — under Production OU

Terraform — global/identity-center

[ ] Zoho SAML metadata URL obtained from Zoho Directory → terraform.tfvars
[ ] terraform apply — SSO wired, 3 permission sets created (Admin, Developer, ReadOnly)
[ ] Groups created: platform-engineers, developers, ops-finance
[ ] Account assignments: platform-engineers = admin in preprod + production
[ ] Delete terraform-admin IAM user key after SSO is verified

Terraform — global/iam ✅ complete (2026-06-18)

[x] GitHub Actions OIDC provider arn:aws:iam::631461173580:oidc-provider/token.actions.githubusercontent.com
[x] finaisse-github-actions role — trusted by all finaisse-org/* repos
[x] terraform-admin IAM user imported into Terraform (terraform_admin.tf, 2026-08-03) — created manually 2026-06-18, now under IaC. Its access key is not managed by Terraform (secret keys cannot be imported); rotation means create-new-via-IaC then delete the old key manually.
[x] sekhar-console human console user (console_user.tf, 2026-08-03) — login profile with password_reset_required, plus inline policy console-access granting sts:AssumeRole on arn:aws:iam::*:role/OrganizationAccountAccessRole (cross-account Switch Role) and iam:ChangePassword on its own ARN (without which the forced first-login reset fails). Password is a sensitive output — read once after apply.

Interim access path. sekhar-console + Switch Role is the stopgap for human console access until IAM Identity Center SSO is wired (global/identity-center above). Revisit both this user and terraform-admin once SSO is verified.

Notes

(Record decisions, blockers, and deviations from the plan here as work proceeds)

Phase 1 — Foundation Infrastructure

Status: ✅ Complete (2026-06-26) — preprod applied
Scope: VPC, RDS, S3, KMS, WAF, Secrets Manager, IAM, VPC Flow Logs
Depends on: Phase 0 complete (preprod account must exist)

What's been Terraformed

Module	What it provisions
`modules/kms`	CMKs for RDS, S3, Secrets Manager — auto-rotating, CloudTrail auditable
`modules/networking`	VPC 10.1.0.0/16, public/private/db subnets ×2 AZs, IGW, NAT, SGs
`modules/rds`	PostgreSQL 16, CMK encrypted, force_ssl, 7d backup retention (preprod) — ⚠️ bump to PG18 before Phase 2 migration: dumps use `uuidv7()`, a PG18-only built-in (see Phase 2 notes)
`modules/s3`	App data + frontend buckets, SSE, versioning, lifecycle
`modules/secrets`	Secrets Manager — 7 app secrets, CMK encrypted
`modules/iam`	ECS task execution role (pull images, inject secrets) + task role (S3 access)
`modules/waf`	WAFv2 — Common rules, KnownBadInputs, SQLi rules, 1000 req/5min rate limit
`modules/flow-logs`	VPC Flow Logs → S3, 90d retention (preprod)
`modules/rds-autostop`	Lambda + EventBridge (12h) — re-stops RDS preprod after AWS 7-day auto-restart

Preprod environment settings (`locals.tf`)

Setting	Preprod	Production
VPC CIDR	10.1.0.0/16	10.0.0.0/16
RDS instance	db.t3.small	db.t3.medium
RDS storage	20 GB	100 GB
RDS Multi-AZ	No	Yes
RDS deletion protection	No	Yes
RDS backup retention	7 days	30 days
WAF rate limit	1000 req/5min	500 req/5min
Flow log retention	90 days	365 days

Outputs (preprod)

Resource	Value
VPC	`vpc-03af2e1cb93c624a1`
RDS endpoint	`finaisse-preprod.cxacwseokgi8.ap-south-1.rds.amazonaws.com:5432`
App data bucket	`finaisse-preprod-app-data`
Frontend bucket	`finaisse-preprod-frontend`
ECS task execution role	`arn:aws:iam::219366469126:role/finaisse-ecs-task-execution-preprod`
ECS task role	`arn:aws:iam::219366469126:role/finaisse-ecs-task-preprod`

Checklist

[x] assume_role set in environments/preprod/providers.tf — preprod account 219366469126
[x] terraform.tfvars filled in (gitignored, local only)
[x] terraform init && terraform plan
[x] terraform apply — 69 resources created
[ ] Verify RDS endpoint reachable from within VPC (Phase 2 task)
[ ] Verify Secrets Manager secrets created ✅ (visible in AWS console)
[ ] Verify WAF web ACL created (ALB association done in Phase 3)
[ ] Verify Flow Logs S3 bucket receiving logs

Estimated monthly cost (Phase 1 only, preprod)

Resource	Cost
RDS db.t3.small	~$25/month
NAT Gateway	~$35/month
KMS (3 CMKs)	~$3/month
WAF	~$5/month
S3 + Flow Logs	~$3/month
GuardDuty	~$3/month
CloudTrail	Free (management events)
Secrets Manager (7 secrets)	~$2/month
Total	~$76/month

Notes

Cost management (preprod idle): NAT Gateway destroyed and RDS stopped to save ~$60/month while not actively testing. To resume: aws rds start-db-instance --db-instance-identifier finaisse-preprod --region ap-south-1 --profile finaisse-mgmt then terraform apply to recreate NAT GW.

RDS auto-restart mitigation (2026-08-03): AWS hard-restarts any stopped RDS instance after 7 days with no API override. modules/rds-autostop deploys a Lambda + EventBridge rule (rate(12 hours)) that checks the instance status and re-stops it if available. This keeps preprod RDS off automatically — no manual intervention needed. The Lambda is wired in environments/preprod/main.tf via module.rds_autostop. CloudWatch logs at /aws/lambda/finaisse-preprod-rds-autostop.

Phase 2 — Data Migration

Status: Not started
Scope: Railway PostgreSQL → RDS, RustFS → S3
Depends on: Phase 1 complete

Checklist

[ ] Dump Railway PostgreSQL (finance + mgmt databases)
[ ] Restore to RDS, validate schema and row counts
[ ] Run prisma migrate deploy against RDS — confirm no drift
[ ] Migrate RustFS bucket contents to S3
[ ] Validate file access via updated env vars pointing to S3
[ ] Run Railway + RDS in parallel briefly (read-only validation period)

Notes

(Record decisions, blockers, and deviations from the plan here as work proceeds)

⚠️ Postgres version: dumps require PG18 on the target — RDS module is currently PG16. Finaisse backups come from a PostgreSQL 18 source and use DEFAULT uuidv7() on primary-key columns. uuidv7() is a built-in function in PG18 only — it does not exist in 16 or 17. Restoring a current dump into a < 18 server fails on every CREATE TABLE with ERROR: function uuidv7() does not exist, and with pg_restore --exit-on-error aborts after the schema with 0 rows loaded (observed 2026-06-19 restoring into Railway staging, which is PG17).

Implications for this phase:

The modules/rds version (PostgreSQL 16, see Phase 1) must be bumped to 18 before the Railway → RDS migration, or the restore will fail identically.
Workaround if a target must stay < 18: pre-install a uuidv7() shim into pg_catalog (so the dump's unqualified call resolves under pg_restore's empty search_path) and restore without --exit-on-error (benign "already exists" collisions otherwise abort the run). Reference implementation: fin-infra/railway/scripts/restore.sh.
Always validate restores by real row counts (SELECT sum(n_live_tup)), not "pg_restore finished" — a uuidv7()-aborted restore leaves a complete-looking schema with no data.

Presigned URLs are not browser-usable on Railway today. The backend can presign (/api/fs/v1/files/contents/presign/:bucket/*, libs/fs/src/fs.service.ts), but URLs are signed against the client's endpoint — http://s3.railway.internal:9000 — which a browser cannot resolve. The UI currently sidesteps this by streaming bytes through blitz-api (POST/GET /api/fs/v1/files/contents/... with Bearer auth), never consuming presigned URLs directly. On AWS this changes: S3 presigned URLs point at a real, browser-reachable host, so we can move uploads/downloads to direct presigned PUT/GET and stop proxying large files through the API. When wiring that, confirm the S3 client endpoint/region used for signing matches the public bucket host (no forcePathStyle hostname mismatch) — otherwise presigned links 403.

Phase 3 — Compute

Status: ✅ Complete (2026-07-02) — preprod applied
Scope: ECS Fargate cluster, ALB, ACM wildcard cert, ElastiCache Valkey, SSM, Cloudflare DNS
Depends on: Phase 1 complete (Phase 2 data migration happens after services are validated)

What was built

Module	What it provisions
`modules/ecs/cluster.tf`	ECS cluster `finaisse-preprod`, Container Insights, CloudWatch log groups (30d), GHCR pull secret in Secrets Manager
`modules/ecs/alb.tf`	ALB (public subnets), HTTP→HTTPS redirect, HTTPS listener (TLS 1.3), WAF association, ACM wildcard cert `.finaisse.com`, Cloudflare DNS validation, wildcard CNAME `.finaisse.com` → ALB
`modules/ecs/services.tf`	10 ECS services: 2 ALB-exposed, 4 internal HTTP, 4 workers; task definitions for all; GHCR image pull via `repositoryCredentials`
`modules/elasticache/`	ElastiCache Valkey 7.2, single node (preprod), `cache.t3.micro`, at-rest encryption
`modules/ssm/`	SSM Parameter Store — non-secret config: node-env, jwtname, temporal config, redis-url, blob host/port, Google Cloud project

Service architecture

Services split into three groups to avoid Terraform target-group-without-ALB errors:

Group	Services	ALB exposure
`alb_services`	blitz-api (10001), blitz-ws (10002)	Path-based routing via ALB
`internal_services`	blitz-agents (10013), blitz-excelrw (10014), blitz-recon (10015), blitz-remotecontrol (10016)	VPC-internal only
`worker_services`	blitz-emailprocessor, blitz-wfw, blitz-wfwpdf, blitz-classicml	No port — `desired_count=0` until Temporal Cloud

ALB routing (path-based):

Priority 10: /api/ws* → blitz-ws target group
Priority 20: /api* → blitz-api target group
Default: 503 "Frontend not yet deployed (Phase 5)"

ACM cert: *.finaisse.com with finaisse.com SAN — both SANs share one CNAME validation record in Cloudflare (Cloudflare de-duplicates them). The wildcard covers all tenant subdomains (acme.finaisse.com, etc.) — no per-tenant cert needed.

Tenant resolution: blitz-api resolves tenants by full-hostname match (Host header → tenant.domainname DB column) — not subdomain parsing. ALB forwards the original Host header unchanged.

GHCR image pull: repositoryCredentials → Secrets Manager secret finaisse/ghcr-pull-secret (username + PAT). Images: ghcr.io/finaisse-org/<service>:latest.

Worker services at desired_count=0: Railway Temporal is .railway.internal — not reachable from the AWS VPC. Workers will be scaled up after Temporal Cloud is configured in Phase 4.

Outputs (preprod)

Resource	Value
ECS cluster	`finaisse-preprod`
ALB DNS name	`finaisse-preprod-1644844663.ap-south-1.elb.amazonaws.com`
ElastiCache endpoint	`finaisse-preprod.2r7obj.ng.0001.aps1.cache.amazonaws.com`
Wildcard DNS	`*.finaisse.com` → ALB (Cloudflare CNAME, `proxied=false`)

Checklist

ECS cluster

[x] ECS cluster finaisse-preprod (Fargate)
[x] Container Insights enabled
[x] CloudWatch log groups per service (30-day retention)
[x] GHCR pull credentials in Secrets Manager

Task definitions

[x] blitz-api — 512 MB, port 10001
[x] blitz-ws — 512 MB, port 10002
[x] blitz-agents — 512 MB, port 10013
[x] blitz-excelrw — 512 MB, port 10014
[x] blitz-recon — 512 MB, port 10015
[x] blitz-remotecontrol — 512 MB, port 10016
[x] blitz-emailprocessor — 512 MB, worker
[x] blitz-wfw — 1024 MB, worker (min 1 GB — OOM observed at 512 MB)
[x] blitz-wfwpdf — 512 MB, worker
[x] blitz-classicml — 512 MB, worker

ALB + DNS

[x] ALB in public subnets
[x] ACM wildcard cert *.finaisse.com (covers all tenant subdomains)
[x] HTTP → HTTPS redirect
[x] HTTPS listener with TLS 1.3 (ELBSecurityPolicy-TLS13-1-2-2021-06)
[x] Target groups for blitz-api and blitz-ws only (internal services have no ALB exposure)
[x] Path-based listener rules (ws priority 10, api priority 20)
[x] WAF web ACL associated with ALB
[x] Cloudflare wildcard CNAME *.finaisse.com → ALB (proxied=false)

ElastiCache

[x] ElastiCache Valkey 7.2, cache.t3.micro, single node (preprod)
[x] At-rest encryption enabled; transit encryption disabled (redis:// — not rediss://)
[x] REDIS_URL passed to all ECS tasks from ElastiCache output

Go-live attempt — 14 September 2026

Preprod was applied. 150 resources created; the environment exists and is reachable. What landed: VPC + NAT, RDS (PostgreSQL 18, replaced), ALB + HTTPS listener + WAF, ElastiCache, the db-backup bucket, the SSM bastion, and all 12 ECS services with their task definitions and log groups.

No container has ever started. Every task failed with CannotPullContainerError: 403 Forbidden — the ghcr_pat in terraform.tfvars is expired (GitHub returns 401 for it directly). Services were scaled to desired_count = 0 to stop the retry loop, which re-pulls the image on every attempt through the NAT gateway.

That is what prompted the move to ECR: with images pulled via the task execution role there is no PAT to expire, and layer traffic stays in-network.

Three apply-time defects were found and fixed in the process, none of which a plan would have surfaced:

Symptom	Cause
`expected DNS record to not already be present`	ACM emits ONE shared validation record for the apex and its first-level wildcard, but `for_each` was keyed on `domain_name` — two resources writing the same DNS name (#230)
`Duplicate secret names found: MGMT_API_KEY`	Listed in both `common_secrets` and apimgmt's `extra_secrets`; the lists concatenate (#230)
`No Fargate configuration exists for given values: 1024 CPU, 1024 memory`	Fargate accepts only fixed pairings; 1 vCPU requires ≥ 2 GB. A `check` block now catches this at plan time (#231)

Also note the RDS instance was replaced, not upgraded, to reach PG18: allow_major_version_upgrade was unset and the instance was stopped, so an in-place major upgrade would have been rejected by the AWS API despite a clean plan.

Validation (pending — blocked on ECR seed + database bootstrap)

[ ] Start RDS (aws rds start-db-instance --db-instance-identifier finaisse-preprod --region ap-south-1 --profile finaisse-mgmt) — note: the rds-autostop Lambda will re-stop it within 12h if you forget
[ ] Confirm NAT Gateway exists (destroyed for cost savings — recreate with terraform apply if needed)
[ ] Seed ECR from GHCR (aws/scripts/sync-ghcr-to-ecr.sh --tag latest) — repositories are created by Terraform but empty
[ ] Create + migrate the mgmt database over the bastion — apimgmt hard-blocks restoring its own control-plane DB, and listing restore targets needs a working mgmt, so the console cannot bootstrap itself
[ ] Restore finance via the mgmt console, then verify by row counts (never "pg_restore finished" — an aborted restore leaves a complete-looking empty schema)
[ ] Repoint the stale tm.tenant connection strings (the mgmt dump is from Dec 2025 and points at Railway)
[ ] All 8 HTTP ECS services running and healthy (images pulled from ECR via the S3 gateway endpoint)
[ ] Test: curl https://<tenant>.finaisse.com/api/health → 200
[ ] Test: WebSocket upgrade at wss://<tenant>.finaisse.com/api/ws
[ ] Run parallel with Railway before traffic cutover

Estimated monthly cost (Phase 3 additions, preprod)

Resource	Cost
ECS Fargate (6 HTTP services × 0.25 vCPU / 512 MB)	~$30/month
ALB	~$20/month
ElastiCache cache.t3.micro	~$15/month
ACM certificate	Free
Phase 3 total	~$65/month
Running total (Phase 1 + 3)	~$141/month

Decisions made

Decision	Rationale
Path-based ALB routing (not host-based)	Matches current Railway nginx pattern — zero frontend changes needed
Wildcard `*.finaisse.com` ACM cert	Covers all tenant subdomains without per-tenant cert management
Internal services have no ALB exposure	agents, excelrw, recon, remotecontrol are called by blitz-api within VPC only
Worker `desired_count=0`	Railway Temporal (`.railway.internal`) unreachable from AWS VPC — unblock in Phase 4 with Temporal Cloud
ElastiCache pulled into Phase 3	Services need `REDIS_URL` to start — can't defer to Phase 4
`proxied=false` on Cloudflare wildcard	ALB handles TLS via ACM; WAF is on the ALB — Cloudflare proxying would double-encrypt

Phase 4 — Workflows

Status: Not started
Scope: Temporal Cloud — connect worker services, scale up from desired_count=0
Depends on: Phase 3 complete
Note: ElastiCache was pulled into Phase 3 (services needed REDIS_URL to start).

Checklist

[ ] Sign up for Temporal Cloud, create namespace finaisse-preprod
[ ] Update temporal_host in aws/environments/preprod/terraform.tfvars (or SSM) — Temporal Cloud endpoint
[ ] Run terraform apply — SSM parameter /finaisse/preprod/temporal-host updates, ECS tasks redeploy
[ ] Scale workers: set worker_desired_count = 1 in environments/preprod/main.tf → terraform apply
[ ] Validate wfw task queues (system, closehub, cash, coresystem, invoice)
[ ] Validate wfwpdf PDF task queue
[ ] Validate classicml ML task queues
[ ] Validate emailprocessor email task queue

Notes

(Record decisions, blockers, and deviations from the plan here as work proceeds)

Phase 5 — Frontend

Status: Not started
Scope: Module Federation build + S3/CloudFront + Cloudflare DNS
Depends on: Phase 3 complete (CloudFront URLs needed before build)

Checklist

[ ] S3 bucket for frontend assets, path-based prefixes per sub-app
[ ] CloudFront distribution with path-based origin routing
[ ] ACM certificate for app.finaisse.com (us-east-1 — required for CloudFront)
[ ] Determine final CloudFront URLs for all 6 sub-apps
[ ] Set VITE_*_REMOTE_URL env vars for all 5 remotes
[ ] Build and deploy 5 remote apps to S3 first
[ ] Build and deploy ui host app with remote URLs baked in
[ ] Cloudflare DNS: app.finaisse.com → CloudFront distribution
[ ] WAF attached to CloudFront distribution

Notes

(Record decisions, blockers, and deviations from the plan here as work proceeds)

Phase 6 — Cutover

Status: Not started
Depends on: Phases 3–5 complete + Sentry integrated in blitz/blitz-ui

Pre-cutover gate — all must be true

[ ] All ECS services healthy for 48+ hours under test traffic
[ ] Temporal Cloud workflows validated end-to-end
[ ] Frontend loading correctly from CloudFront
[ ] Sentry integrated in all services
[ ] CloudWatch alarms configured and tested
[ ] Rollback plan documented and rehearsed

Checklist

[ ] Cloudflare DNS: api.finaisse.com → ALB
[ ] Monitor for 1 week — error rates, latency, Sentry alerts
[ ] Decommission Railway production services (Railway moves to dev-only)
[ ] Archive Railway IaC (keep fin-infra/railway/ for staging/dev)

Notes

(Record decisions, blockers, and deviations from the plan here as work proceeds)

Phase 7 — Observability

Status: Not started

Checklist

[ ] CloudWatch Log Groups per service (verify retention set to 30 days)
[ ] Container Insights enabled on ECS cluster
[ ] CloudWatch alarms: CPU > 80%, memory > 80%, ALB 5xx rate > 1%
[ ] SNS ops alerts already configured (Phase 0) — wire CloudWatch alarms to it
[ ] Sentry organisation created
[ ] @sentry/node added to all Bun/Elysia services
[ ] @sentry/vue added to blitz-ui shell app
[ ] sentry-sdk added to blitz-classicml
[ ] SENTRY_DSN in Secrets Manager, injected into all ECS task definitions

Notes

(Record decisions, blockers, and deviations from the plan here as work proceeds)

Future — Cloudflare IaC

Status: Started (2026-09-11). cloudflare/ Terraform root exists in fin-infra alongside aws/, with a phased adoption plan documented in its own cloudflare/README.md. Most of Cloudflare (DNS, Zero Trust, Gateway policies) is still dashboard-managed — only the piece below is live in Terraform. This checklist mirrors the README's phases; update both together.

[x] Add cloudflare/ Terraform root to fin-infra (fin-infra#98, merged 2026-09-11)
[x] Phase 3 (partial) — Access application + policy for temporal.finaisse.com (GitHub-org SSO), closing fin-infra#195. Live in state: cloudflare_access_application.temporal_ui, cloudflare_access_policy.temporal_ui_github_org.
[ ] Phase 1 — Import existing DNS records (scaffolding exists in dns_railway.tf/dns_pages.tf; imports_completed not yet flipped, no records actually imported)
[ ] Phase 2 — Import Pages projects
[ ] Phase 3 (remainder) — Import the other Access applications (internal-docs, mgmt UI) — drafted in cloudflare/access_existing.tf, not yet applied
[ ] Phase 4 — Import Zero Trust / Gateway policies
[ ] Phase 5 — AWS-side Cloudflare records (ALB/ACM), once AWS goes live

Tracked in fin-infra#203 — reference it (and cloudflare/README.md for phase order and blast-radius notes) when this is picked back up.

Future — SOC 2 / Compliance Readiness

When a compliance requirement arrives, re-enable in a single Terraform PR:

[ ] Enable AWS Config recorder (control-tower/main.tf — flip is_enabled = false → true)
[ ] Enable Security Hub (aws_securityhub_account resource, org-wide)
[ ] Enable CloudTrail data events (S3 reads/writes)
[ ] Enable GuardDuty malware protection + EKS audit logs if applicable
[ ] Penetration test (required for most frameworks)
[ ] Data classification policy documented
[ ] Incident response runbook reviewed and rehearsed
[ ] Backup restore tested and documented

Decision Log

Date	Decision	Rationale
2026-06-04	Terraform over CDK	Multi-cloud scope: AWS + Cloudflare + future GCP/Gemini
2026-06-04	RabbitMQ removed	All async workflows are Temporal-managed
2026-06-04	GHCR retained	No benefit to migrating to ECR
2026-06-16	16 services confirmed	4 infra (→ AWS managed) + 12 ECS app services (as of 2026-09-15; `bapiproxy` landed later) + frontends
2026-06-16	Sentry for error tracking	No error tracking exists today; CloudWatch alone insufficient
2026-06-17	ap-south-1 (Mumbai)	Data residency, RBI alignment, full service catalog; Hyderabad rejected (missing ElastiCache/CloudFront)
2026-06-17	Control Tower + AFT	Governance from day 1; retrofitting at 5+ accounts is painful (prior experience)
2026-06-17	Zoho SSO over GitHub SSO	GitHub SAML requires Enterprise plan; Zoho already in use and covers all staff
2026-06-17	Security baseline in Phase 0/1	KMS CMKs, WAF, GuardDuty, SCPs pulled forward — not deferred to Phase 8
2026-06-17	AWS Config disabled initially	Re-enable at SOC 2 prep; saves ~$5-8/month until needed
2026-06-17	All resources via Terraform	No manual console provisioning; two unavoidable exceptions: IAM Identity Center enable + Control Tower wizard
2026-06-17	Production region: us-east-1	Future — not started; preprod in Mumbai (ap-south-1) first
2026-07-02	Path-based ALB routing	Matches current Railway nginx pattern — no frontend changes needed
2026-07-02	Wildcard `*.finaisse.com` ACM cert	Covers all tenant subdomains; both SANs share one Cloudflare CNAME validation record
2026-07-02	Internal services not ALB-exposed	agents, excelrw, recon, remotecontrol are VPC-internal; only api + ws face the internet
2026-07-02	Worker `desired_count=0`	Railway Temporal not reachable from AWS VPC — unblock after Temporal Cloud (Phase 4)
2026-07-02	ElastiCache moved to Phase 3	Services need `REDIS_URL` at startup; cannot defer to Phase 4
2026-07-02	Cloudflare `proxied=false` on wildcard	ALB terminates TLS via ACM; WAF on ALB — Cloudflare proxy would break certificate chain
2026-08-03	RDS autostop Lambda	AWS platform auto-restarts stopped RDS after 7 days with no API override; Lambda + EventBridge every 12h is the standard workaround
2026-08-03	`sekhar-console` IAM user for console access	Root cannot Switch Role to member accounts; IAM user with `sts:AssumeRole` on `OrganizationAccountAccessRole` is the interim path until IAM Identity Center SSO is configured
2026-09-14	ECR replaces GHCR as the ECS pull source	Reverses the 2026-06-04 "GHCR retained" row, which carried no recorded rationale and predates AWS being live. Fargate is in private subnets, so a GHCR pull crosses the NAT gateway (~264 MB/image, ~3.2 GB per 12-service cycle). ECR pulls use the task execution role — no PAT to expire, which had just taken the cluster down. GHCR stays the build output; `aws/scripts/sync-ghcr-to-ecr.sh` syncs
2026-09-14	S3 gateway VPC endpoint only, no interface endpoints	ECR stores layers in S3 and redirects there, so the free gateway carries ~all the bytes. An interface endpoint is ~$7.30/mo fixed and needs ~47 GB/mo to break even; `ecr.api`+`ecr.dkr` would need ~165,000 pull cycles/mo against ~100 realistic. Reconsider `logs` at production scale
2026-09-14	`FARGATE_SPOT` for preprod, `FARGATE` pinned for production	~70% cheaper; a 2-minute reclaim is fine for a POC and an outage in production. Note services had `launch_type` hardcoded, which silently overrides the cluster default strategy — had to move to `capacity_provider_strategy`
2026-09-14	All tasks at the Fargate floor (0.25 vCPU / 512 MB)	Deliberately NOT Railway parity, which is higher throughout and would roughly double cost with no load. Raise on evidence. ⚠️ `blitz-api` (1 GB on Railway, OOM-confirmed at 512 MB) and `wfw` (1.5 GB) are expected to need it first
2026-09-14	RDS on PostgreSQL 18, instance replaced not upgraded	The `.bkp` dumps are PG18 and use `DEFAULT uuidv7()`, a PG18-only builtin — restoring into PG16 fails on every CREATE TABLE. Railway works around this with a `pg_catalog` shim; moving to 18 removes the need. Replaced rather than upgraded because the instance was empty and stopped
2026-09-14	Count-gated SSM bastion for operator DB access	RDS is private-subnet with 5432 from the ECS SG only, so there was no path from a laptop. A `t3.nano` with ZERO ingress (SSM agent polls outbound), IAM-gated and CloudTrail-logged. On the critical path for bring-up: `mgmt` cannot be created through the mgmt console
2026-09-14	Cost guardrails: circuit breaker + NAT egress alarm	ECS has no max-retries for task placement, so a failed pull retries forever and re-pulls each time — this has cost ~$100 once. Budgets evaluate daily and lag; the NAT alarm is minute-resolution. The circuit breaker only helps a deployment in progress, not a never-succeeded first rollout
2026-09-25	Admin console moved to `mgmt-preprod.finaisse.com` (first-level), gated by Cloudflare Access on a GitHub team	It was `mgmt.app.finaisse.com`, served by the DNS-only `.app` wildcard, so Cloudflare never saw the traffic and Access could not gate it: the console loaded and every `/api` call 403'd on the nginx `Cf-Access-Jwt-Assertion` check, leaving that header the only defence. Proxying `mgmt.app` cannot fix it* - Cloudflare's free Universal SSL covers `.finaisse.com`, exactly ONE label, so the edge has no certificate for a second-level name and aborts the handshake (verified: `tls alert handshake failure`, `no peer certificate available`); that needs Advanced Certificate Manager (~$10/mo). A first-level host is covered at the edge, and the ALB's ACM cert already carries `.finaisse.com` as a SAN so the origin hop validates too - $0, and no blitz-ui change, since traffic still arrives via Cloudflare Access and the existing nginx gate keeps working. The `.app` tenant wildcard stays DNS-only: that keeps AWS WAF the only WAF layer by design, and keeps the login rate-limit rules meaningful (they aggregate on the CONNECTION IP, so proxying would collapse every client into Cloudflare's addresses). Authorisation is GitHub team* `fin-be-devs`, not org membership - the org has 9 members and this console reaches tenant DB connection strings, so a `github-organization` include would silently widen access to 9. ALB `authenticate-oidc` was considered and rejected for now: it needs nginx to read `x-amzn-oidc-data`, and GitHub cannot back it at all (`github.com` serves no OIDC discovery document and issues OAuth 2.0 access tokens, not OIDC ID tokens). fin-infra#260, #286

Run & Operate

Recovery Procedures

Frontend

AWS Production Implementation Log ​

Terraform Root Structure ​

Apply order (first-time setup) ​

Phase 0 — Account Foundation ​

Decisions made ​

Checklist ​

One-time manual prerequisites ​

Terraform — global/bootstrap ​

Terraform — provider lock files (2026-09-24) ​

Terraform — global/control-tower ✅ complete (2026-06-18) ​

Workload accounts (direct, AFT removed) ​

Terraform — global/identity-center ​

Terraform — global/iam ✅ complete (2026-06-18) ​

Notes ​

Phase 1 — Foundation Infrastructure ​

What's been Terraformed ​

Preprod environment settings (locals.tf) ​

Outputs (preprod) ​

Checklist ​

Estimated monthly cost (Phase 1 only, preprod) ​

Notes ​

Phase 2 — Data Migration ​

Checklist ​

Notes ​

Phase 3 — Compute ​

What was built ​

Service architecture ​

Outputs (preprod) ​

Checklist ​

ECS cluster ​

Task definitions ​

ALB + DNS ​

ElastiCache ​

Go-live attempt — 14 September 2026 ​

Validation (pending — blocked on ECR seed + database bootstrap) ​

Estimated monthly cost (Phase 3 additions, preprod) ​

Decisions made ​

Phase 4 — Workflows ​

Checklist ​

Notes ​

Phase 5 — Frontend ​

Checklist ​

Notes ​

Phase 6 — Cutover ​

Pre-cutover gate — all must be true ​

Checklist ​

Notes ​

Phase 7 — Observability ​

Checklist ​

Notes ​

Future — Cloudflare IaC ​

Future — SOC 2 / Compliance Readiness ​

Decision Log ​

AWS Production Implementation Log

Terraform Root Structure

Apply order (first-time setup)

Phase 0 — Account Foundation

Decisions made

Checklist

One-time manual prerequisites

Terraform — global/bootstrap

Terraform — provider lock files (2026-09-24)

Terraform — global/control-tower ✅ complete (2026-06-18)

Workload accounts (direct, AFT removed)

Terraform — global/identity-center

Terraform — global/iam ✅ complete (2026-06-18)

Notes

Phase 1 — Foundation Infrastructure

What's been Terraformed

Preprod environment settings (`locals.tf`)

Outputs (preprod)

Checklist

Estimated monthly cost (Phase 1 only, preprod)

Notes

Phase 2 — Data Migration

Checklist

Notes

Phase 3 — Compute

What was built

Service architecture

Outputs (preprod)

Checklist

ECS cluster

Task definitions

ALB + DNS

ElastiCache

Go-live attempt — 14 September 2026

Validation (pending — blocked on ECR seed + database bootstrap)

Estimated monthly cost (Phase 3 additions, preprod)

Decisions made

Phase 4 — Workflows

Checklist

Notes

Phase 5 — Frontend

Checklist

Notes

Phase 6 — Cutover

Pre-cutover gate — all must be true

Checklist

Notes

Phase 7 — Observability

Checklist

Notes

Future — Cloudflare IaC

Future — SOC 2 / Compliance Readiness

Decision Log