Incident Response Runbook

Severity Levels

Level	Description	Target Response
P1	Production down, data loss risk	Immediate
P2	Production degraded, major feature broken	< 1 hour
P3	Non-critical feature broken	Next business day

P1 Response Steps

1. Assess — check Railway dashboard and service logs
2. Communicate — notify team via Slack/Zoho
3. Mitigate — if a recent deploy is the cause, roll back immediately (see Rollback)
4. Investigate — pull logs: bun run logs
5. Resolve — fix forward or roll back
6. Post-mortem — document in incidents/ folder in fin-infra

P2 Response Steps

1. Assess — identify affected service(s) and scope of impact
2. Communicate — notify team; set expectations on resolution time
3. Triage — determine if a hotfix or rollback is faster
4. Resolve — deploy fix or roll back
5. Document — note what happened in the related issue or PR

Useful Commands

# Check all service statuses
bun run status

# Tail logs for all services
bun run logs

# Tail logs for a specific service
bun run logs -- --service blitz-api
bun run logs -- --service ws
bun run logs -- --service wfw

# Redeploy a specific service
bun run scripts/deploy.ts blitz-api

All commands run from fin-infra/railway/.

Key Resources

Resource	Link
Railway dashboard	https://railway.app
GitHub Actions	https://github.com/finaisse-org/fin-infra/actions
GHCR images	https://github.com/orgs/finaisse-org/packages
blitz repo	https://github.com/finaisse-org/blitz

Common Failure Patterns

Service OOM (Out of Memory)

• wfw requires at least 1 GB RAM. If it crashes immediately after deploy, check memory allocation in the Railway dashboard.

Frontend not loading

• Check if blitz-api is healthy — the frontend depends on it
• Verify Module Federation remote URLs are reachable (all 6 sub-apps must be up)

Database connection errors

• Check DATABASE_URL and MANAGEMENT_DATABASE_URL are correctly set
• Verify PostgreSQL service is healthy in Railway dashboard

Workflow failures

• Check Temporal UI for stuck or failed workflows
• Check wfw logs for worker errors