Run & Operate

Day-to-day operations for the Finaisse/Blitz platform.

Railway

Key External Links

Resource	URL
Demo Tenant App (frontend)	https://letsgo.finaisse.com
Railway dashboard	https://railway.app
GitHub Actions	https://github.com/finaisse-org/fin-infra/actions
GHCR images	https://github.com/orgs/finaisse-org/packages
Internal docs	https://internal-docs.finaisse.com

Staging backends (blitz-api, ws, wfw, …) are private on *.railway.internal — no public URL; the frontend's nginx proxies /api to them.

Scaling Staging Up/Down

Staging auto-scales down every night at 10:00 PM IST (cost saving) via the scheduled scale-staging.yml workflow. To scale back up in the morning — or scale down early:

Via GitHub Actions (recommended):

1. Go to fin-infra Actions
2. Select Scale Staging
3. Run workflow → choose up or down and the target (all / backend / frontend / infra)

Workflow	Trigger	Purpose
scale-staging.yml	Manual + daily schedule	Scale staging up/down; auto-downs at 10:00 PM IST

Via terminal:

bun run scale up all      # Scale everything back up
bun run scale down all    # Scale everything down

Checking Service Health

bun run status

Also check the Railway dashboard at railway.app — the Deployments tab shows health check status per service.

Viewing Logs

# All services
bun run logs

# Specific service
bun run logs -- --service blitz-api
bun run logs -- --service ws
bun run logs -- --service wfw

Incidents

Severity definitions and response times are set by the Incident Response Policy — that page owns them; this runbook covers the mechanics only. Note a suspected compromise of customer Restricted data is always P1 regardless of service impact.

P1 response:

1. Assess — check Railway dashboard and service logs
2. Communicate — notify team via Slack/Zoho
3. Mitigate — if a recent deploy is the cause, roll back immediately (see Rollback)
4. Investigate — pull logs: bun run logs
5. Resolve — fix forward or roll back
6. Post-mortem — blameless review for every P1 and significant P2; findings become tracked issues in the Findings Register, per the Incident Response Policy. (This previously said "document in incidents/ folder in fin-infra" — that folder has never existed, and it contradicted the policy's evidence destination.)

P2 response:

1. Assess — identify affected service(s) and scope of impact
2. Communicate — notify team; set expectations on resolution time
3. Triage — determine if a hotfix or rollback is faster
4. Resolve — deploy fix or roll back
5. Document — note what happened in the related issue or PR

Common failure patterns:

• Service OOM — wfw requires at least 1 GB RAM. If it crashes immediately after deploy, check memory allocation in the Railway dashboard.
• Frontend not loading — check if blitz-api is healthy (the frontend depends on it), and that Module Federation remote URLs are reachable (all 6 sub-apps must be up).
• Database connection errors — check DATABASE_URL / MANAGEMENT_DATABASE_URL are correctly set, and that PostgreSQL is healthy in the Railway dashboard.
• Workflow failures — check Temporal UI for stuck or failed workflows, and wfw logs for worker errors.

Recovery Procedures

For deeper data-loss recovery beyond a service rollback:

• Staging DB Restore Console — refresh the staging finance database from a backup .bkp
• RustFS Bucket Recovery — recreate object-storage buckets after a deliberate store loss

Pushing a Variable Change

1. Edit railway/shared/variables.ts in fin-infra
2. Run bun run vars:sync to push to all backend services
3. Services may need redeployment to pick up the new values: bun run deploy:backend

Provisioning

To provision or re-provision infrastructure:

bun run provision:dry       # Preview what would change
bun run provision           # Provision everything (idempotent)
bun run provision:infra     # Infra only (Postgres, RustFS, Valkey, Temporal)
bun run provision:services  # Microservices + frontend only

Initialising a New Database

Run once when setting up a new environment:

bun run init-db    # Creates finance and mgmt databases

Useful Commands (reference)

All commands run from the railway/ directory in fin-infra:

bun run status          # Show deployment status of all services
bun run logs            # View service logs (all)
bun run logs -- --service blitz-api   # Logs for a specific service
bun run deploy:all      # Redeploy all services
bun run vars:sync       # Push environment variables to Railway services

Local

See Local Development for running and operating the full stack locally.

AWS

AWS operations are not yet applicable — production traffic does not run on AWS today. See AWS Implementation for current status.