RustFS Bucket Recovery Runbook

Recreate the RustFS/S3 buckets after staging's object store has been wiped.

Symptom: file upload fails with a 500, e.g. the Setup Scripts screen (Settings → Dev Tools → Setup Scripts) returns [POST] "/api/fs/v1/files/contents/temp/masters/": 500.

The recurring wipe was FIXED on 2026-08-26

RustFS now has a working persistent volume, so the object store survives redeploys and the nightly scale-down. This is no longer a daily chore.

You should only need this runbook after a deliberate loss of the object store — recreating the service, detaching/replacing the volume, or standing up a new environment. If buckets vanish on an ordinary redeploy or scale-up, that is a regression: check the volume first (see Verify the volume is actually working) rather than just re-running the recovery.

TL;DR

cd fin-infra/railway
set -a && . ./.env && set +a

# 1. Recreate the `default`/`temp` buckets (blitz-api does this at startup).
#    Railway dashboard → blitz-api → Redeploy, or via the API.

# 2. Recreate EVERY OTHER tenant's UUID buckets — one workflow run per tenant.
#    Temporal UI → Start Workflow (see fields below).

# 3. Retry the upload.

Verify at any point by listing the live buckets:

railway ssh --project "$RAILWAY_PROJECT_ID" --environment "$RAILWAY_ENVIRONMENT_ID" \
  --service "RustFS/S3" 'ls -1 /data/store | grep -v rustfs.sys'

Service name, not IaC name

railway ssh wants the Railway display name — "RustFS/S3". Passing s3 fails with Service 's3' not found. (The fin-infra scripts handle this via aliases; the CLI does not.)

Step 1 — Redeploy blitz-api

blitz-api calls createDefaultBuckets() at startup, which creates exactly two buckets — the literal default and temp:

// blitz/src/libs/bm/src/bm.service.ts
public async createDefaultBuckets(): Promise<void> {
  await this.createBucket("default", undefined);
  await this.createTempBucket("temp");     // 1-day expiry lifecycle
}

A redeploy re-runs it. Confirm in the logs:

bms creating default buckets...
create bucket...
🦊 api server is running - 0.0.0.0:10001

This alone fixes the default tenant (letsgo.finaisse.com), whose dbucket/ tbucket are the literal names default/temp. Every other tenant is still broken — continue to step 2.

Step 2 — Re-run provisionTenant per tenant

Any tenant created through the mgmt portal gets UUID bucket names, assigned once at creation. Nothing recreates them automatically.

2a. Get the tenant's ID and bucket names

Postgres is private-only, so query it from inside the network:

railway ssh --project "$RAILWAY_PROJECT_ID" --environment "$RAILWAY_ENVIRONMENT_ID" \
  --service "Postgres" \
  'psql "postgresql://$PGUSER:$PGPASSWORD@localhost:5432/mgmt" \
     -c "select id, name, domainname, dbucket, tbucket from tm.tenant order by name;"'

Use the mgmt database explicitly

The Postgres service's own DATABASE_URL points elsewhere — psql "$DATABASE_URL" fails with relation "tm.tenant" does not exist. Build the URL against /mgmt as above.

Example (staging, 2026-08-25):

name	domainname	dbucket	tbucket
default	letsgo.finaisse.com	default	temp
Finaisse Test	test-app.finaisse.com	fd1742fe-…	5c52fb93-…

2b. Start the workflow

Temporal UI (staging: temporal-ui-stage.up.railway.app) → Workflows → Start Workflow:

Field	Value
Workflow ID	provision-<something-unique>
Task Queue	taskqueue-system
Workflow Type	provisionTenant
Data	"d28a5f20-3392-42a8-bf6f-eedeee94eaca"
Encoding	json/plain

Two things that will bite you

1. Workflow Type is a free-text field. Temporal's form does not list registered workflows — there is nothing to pick from. Type provisionTenant exactly.
2. The Data payload is a quoted bare string, not an object. The workflow signature is provisionTenant(tenantId: string) and the caller passes args: [tenant.id]. {"tenantId": "..."} is wrong and fails in LoadTenant.

2c. Confirm the result

The run completes in well under a second. Expected result:

{
  "tenantId": "d28a5f20-3392-42a8-bf6f-eedeee94eaca",
  "dbName": "fintest",
  "created": false,
  "buckets": ["fd1742fe-…", "5c52fb93-…"],
  "enabled": true
}

"created": false is the important field — it means CreateDatabase found the tenant DB already present and skipped it. If you ever see "created": true on a recovery run, the tenant DB was just recreated from the template and any data in it is gone; stop and escalate.

Is re-running provisionTenant safe?

Yes, on an existing tenant. Every step self-guards:

Activity	Behaviour on re-run
LoadTenant	read-only
CreateDatabase	skips if the DB exists (created: false)
CreateBuckets	idempotent — handles BucketAlreadyOwnedByYou
EnableTenant	sets isenabled = true, already true

DropDatabase is a separate activity that this workflow never calls — destroying tenant data has to be deliberate.

Using the real workflow is also better than creating buckets by hand: createTempBucket applies the 1-day expiry lifecycle to tbucket, which a manual create omits.

Step 3 — Retry the upload

Confirm all expected buckets exist, then retry. For staging as of 2026-08-26 that is four (two per tenant):

default                                  ← default tenant  dbucket
temp                                     ← default tenant  tbucket
fd1742fe-cfcf-48a8-90a1-7e13383c51a9     ← Finaisse Test   dbucket
5c52fb93-0cdc-4750-b347-e3c2686f62db     ← Finaisse Test   tbucket

Why this used to happen (fixed 2026-08-26)

A Railway service without a volume keeps everything in the container's ephemeral filesystem, so the object store was lost on every container replacement — every redeploy and every nightly scale-down/up. A volume was declared in fin-infra#79 but could not mount: attaching it crash-looped the service with

[FATAL] Server runtime failed: Io error: Permission denied (os error 13)

Root cause — an image/platform mismatch, not a RustFS bug and nothing to do with tenancy. Railway mounts volumes root-owned (mode 755) and, unlike Docker with an empty named volume, does not initialise the mount's ownership from the image. The upstream image declares USER rustfs (uid 10001), so it cannot write its own data dir. Its entrypoint does contain chown logic (RUSTFS_UID/RUSTFS_GID) but never reaches it — privileges are already dropped. Postgres survives the identical platform only because its image starts as root, chowns PGDATA, then drops via gosu.

The fix (fin-infra#94) is three variables on the s3 service, no custom image:

RAILWAY_RUN_UID=0      # Railway's documented override for a non-root image's USER
RUSTFS_UID=10001       # upstream entrypoint chowns /data to this on first boot
RUSTFS_GID=10001

Railway has no fsGroup equivalent (VolumeCreateInput accepts only projectId, environmentId, serviceId, mountPath); RAILWAY_RUN_UID is the substitute, applied at the process level. Consequence: RustFS runs as root in the container — acceptable because the service is private-only with no public domain.

Do not retry these — all four were tested and fail

• Writing the mount root as uid 10001 → Permission denied.
• A subpath (RUSTFS_VOLUMES=/data/store) → mkdir denied. Creating a child needs write access to the parent, which is the root-owned mount root. (This was documented here as the fix on 2026-08-25 before it was tested. It does not work.)
• chown from the service's own startCommand → Operation not permitted.
• A wrapper image that chowns as root then drops → works, but unnecessary. If ever needed: the image has no gosu, and its setpriv is the busybox build with no --reuid, so busybox su is the only privilege-drop path.

Verify the volume is actually working

Run this if buckets ever disappear unexpectedly. A clean boot does not prove the volume is mounted — check the device IDs:

railway ssh --project "$RAILWAY_PROJECT_ID" --environment "$RAILWAY_ENVIRONMENT_ID" \
  --service "s3" 'stat -c "%n dev=%d" / /data; grep -c "/data" /proc/mounts'

Healthy looks like this — /data on a different device from /, and one mount entry:

/ dev=27263062
/data dev=24176288
1

If both show the same device, the volume is not mounted and the store is ephemeral again. Also look for Mounting volume on: … in the deploy logs; its absence is the tell.

A volume can report healthy while silently not mounting

A volume pending deletion reports state: READY and injects RAILWAY_VOLUME_MOUNT_PATH, yet is never mounted. Check isPendingDeletion, not just state. Railway soft-deletes with a 48-hour grace period during which the slot stays occupied — volumeCreate then fails with "A service can only have one volume", and re-issuing volumeDelete is a no-op. The dashboard's Restore control lives in the Activity feed (it reverts the change patch), not in service Settings.

A rename does not move the private hostname

A Railway service's *.railway.internal hostname is fixed at creation. Renaming the service changes only the display name — after renaming s3new → s3, DNS still answered only on s3new.railway.internal and every BLOB_HOST lookup failed ENOTFOUND. Edit the private domain in the dashboard (Settings → Networking); the API does not expose it.

Backups

Configured 2026-08-26 (fin-infra#95). Both persistent volumes are now on a DAILY + WEEKLY schedule. Check the current state any time:

cd fin-infra/railway
set -a && . ./.env && set +a
bun run backups          # read-only; non-zero exit if any volume has NO schedule

Volume	Daily	Weekly
s3new-volume (RustFS, /data)	19:14 IST	Sat 16:41 IST
postgres-volume	12:58 IST	Sat 07:31 IST

Railway picks the cron minute itself — the tier list is the only input, so those times were assigned, not chosen.

Postgres was effectively unbacked-up for two months

Before this, neither volume had a schedule. Postgres's only backup was a 2026-06-22 side effect of a template auto-update — not a policy. RustFS had none at all, having only gained a working volume the day before.

Retention and purge

Platform-fixed and read-only — volumeInstanceBackupScheduleUpdate accepts only (kinds, volumeInstanceId), with no cron or retention argument:

Tier	Retention
DAILY	6 days (518,400 s)
WEEKLY	27 days (2,332,800 s)
MONTHLY	available, not enabled

Scheduled backups carry an expiresAt and Railway purges them automatically — no cleanup work. volumeInstanceBackupLock pins one past its tier window (e.g. before a risky migration).

Manual snapshots are NEVER purged

A backup created by hand has expiresAt: null and scheduleId: null. It persists until someone calls volumeInstanceBackupDelete, accumulating cost silently — exactly how the June 2026 Postgres backup sat unnoticed. bun run backups flags them. Use a manual snapshot for a deliberate pre-change checkpoint, not for routine cover.

How much gets backed up — not the 50 GB

Backups are incremental / copy-on-write, so the provisioned ceiling is irrelevant to cost. Three size fields, easy to confuse:

Field	Meaning	Measured 2026-08-26
volumeInstanceSizeMB	provisioned ceiling	50,000 MB
referencedMB	logical data the snapshot points at	1,120–1,413 MB
usedMB	blocks the snapshot uniquely owns	0–2 MB at creation

A fresh snapshot shares every block with the live volume, so it costs almost nothing and only grows as the volume diverges — the June Postgres backup reads 352 MB for that reason, while one taken the same instant read 0. A daily schedule therefore bills roughly the daily churn, not 50 GB × retention.

Not verifiable from the API: whether backup storage is billed separately on our plan — the schema exposes sizes but no pricing.

Restore

volumeInstanceBackupRestore(volumeInstanceBackupId, volumeInstanceId, wipeServiceIds, replicaServiceIds)
volumeInstancePITRRestore(targetTimestamp, volumeInstanceId, newServiceName, sourceRepoPath)

This is the supported way to recover object storage. Do not attempt to move data by tarring /data through railway ssh — command-argument size limits make it impractical, and piped stdin loses the project context.

A deleted volume's data is gone even within the grace period

Railway soft-deletes with a 48 h grace, but that does not create a backup. The two orphans left by the 2026-08-25 cutover (rustfs/s3-volume, uidtest-volume) had no snapshot and no schedule, so their contents were unrecoverable once deletion completed. If you may want a detached volume's data back, snapshot it before the grace period closes. bun run backups deliberately skips pending-deletion volumes, so their absence of backups will not show up there.

Related

• Volume converge + rollback: fin-infra/railway/scripts/apply-volumes.ts, detach-volume.ts (bun run volumes, bun run volumes:dry)
• Staging DB Restore Console — the DB half of restoring a tenant
• Tracking: fin-infra#79 (volume declared), fin-infra#94 (made it work), fin-infra#95 (backups + bun run backups), fin-infra#92 (DB var drift), fin-infra#93 (bun run staleness)
• blitz#1353 — only the default tenant's buckets are recreated after a wipe
• blitz#1354 — fileUnzip reports success when no objects were written