Observability & Usage Telemetry — Design
One framework for all operational and usage signals across the platform. Supersedes the per-feature approach that left LLM Token-Usage Capture stalled at "emission target deferred".
Status: design, August 2026. Backend decided (Grafana Cloud); phased delivery below.
Driver: two concrete needs arrived independently — LLM token/cost capture and app-switcher usage analytics — and a third will follow. Each was about to grow its own pipeline. This defines the shared one.
The distinction that shapes everything
"Metrics" gets used for two genuinely different things. Conflating them is the most common and most expensive mistake in this area.
| Operational telemetry | Product / usage analytics | |
|---|---|---|
| Question | Is it up, fast, erroring, costly? | Who used what, how often? |
| Shape | Pre-aggregated numeric time series | One record per occurrence |
| Cardinality | Must stay low | High by design |
| Retention | Weeks–months | Months–years |
| Consumer | On-call, ops | Product, CS, ops |
| Store | Prometheus (via Grafana Cloud) | Postgres → warehouse later |
Our two initial use cases land on opposite sides:
- LLM tokens → operational + cost. Dimensions: tenant, model, agent. Bounded. → metrics.
- App-switch usage → product analytics. Dimensions include user. Unbounded. → events.
The cardinality rule (non-negotiable)
Never put a user ID — or any unbounded value — in a metric label.
Prometheus creates one time series per unique label combination. 10k users × 20 metrics = 200k series, and the backend degrades then falls over. This single mistake accounts for most self-inflicted observability outages in the industry.
Safe labels: environment, tenant, service, app, model, status, route_template. Never labels: user_id, invoice_id, session_id, raw URLs, error messages.
Per-user data is not forbidden — it belongs in the event store, which is built for it.
Architecture
application code
│ emit(...) ← one facade, all services
▼
OpenTelemetry SDK
│ OTLP
▼
OTel Collector ← one deployment; routing lives here
├──────────────► Grafana Cloud Prometheus (metrics)
├──────────────► Grafana Cloud Loki (logs)
├──────────────► Grafana Cloud Tempo (traces)
└──────────────► Postgres (usage events)Emit once, route by type. Call sites never know the backend. Swapping Prometheus for AWS Managed Prometheus, or Postgres for a warehouse, is a collector config change — zero application edits. That property is the whole point of choosing OTel over vendor SDKs.
Why Grafana Cloud
Free tier covers Prometheus + Loki + Tempo with no infrastructure to run, back up, or monitor. Instrumentation is identical if we later self-host or move to AWS Managed Prometheus — the SDK and collector config carry over, only the exporter endpoint changes.
The alternative, self-hosting Prometheus, means Thanos or Mimir for durable long retention. That is real operational load for no benefit at our current scale, and it is reversible later precisely because OTel keeps the call sites clean.
All three pillars
Metrics — numeric aggregates for dashboards and alerts. Logs — structured records; Pino output ships to Loki with trace correlation. Traces — request causality across services. Directly useful here: a request entering blitz-api, dispatching a Temporal workflow, calling an LLM, hitting Postgres, and (for SAP) crossing into bapiproxy is currently opaque. Traces make that one waterfall.
Resource attributes — the dimension hierarchy
Set once per service at startup, attached automatically to every signal:
| Attribute | Example | Purpose |
|---|---|---|
service.name | blitz-api | which service |
service.version | git SHA | correlate with deploys |
deployment.environment | staging | production | environment split |
cloud.provider | railway | aws | platform split |
cloud.region | ap-south-1 |
This answers the "Railway and AWS separately" requirement directly: one backend, filtered by deployment.environment / cloud.provider, with side-by-side comparison in the same dashboard. Environment sits above tenant in the hierarchy:
environment → cloud → service → tenant → app → (user: events only)Signal catalogue
Metrics (Prometheus)
| Metric | Type | Labels |
|---|---|---|
llm.tokens | counter | tenant, model, agent, token_type (input/output/cached/reasoning) |
llm.request.duration | histogram | tenant, model, agent, status |
http.server.duration | histogram | service, route_template, status |
temporal.workflow.duration | histogram | service, workflow_type, status |
app.switch.count | counter | tenant, from_app, to_app, method |
finops.job.terminal | counter | tenant, service, job_type, status (completed/failed) |
finops.task.state | gauge | tenant, service, state (failed/blocked/running/…) |
finops.feed.age_seconds | gauge | tenant, feed |
Note app.switch.count appears here without user — the aggregate adoption view. Per-user detail lives in events.
The finops.* metrics are business-process health — see Domain metrics below.
Usage events (Postgres)
One table, one row per occurrence, for anything user-attributed:
CREATE TABLE usage_event (
id bigserial PRIMARY KEY,
occurred_at timestamptz NOT NULL DEFAULT now(),
environment text NOT NULL,
tenant_id uuid NOT NULL,
user_id uuid,
event_type text NOT NULL, -- 'app_switch' | 'llm_call' | …
attributes jsonb NOT NULL -- type-specific payload
);
CREATE INDEX ON usage_event (tenant_id, event_type, occurred_at DESC);
CREATE INDEX ON usage_event (occurred_at);event_type + jsonb means a new signal is a new event_type, not a migration. Adding the third use case costs nothing structurally.
Daily rollups for dashboards, so queries never scan raw events:
CREATE TABLE usage_daily (
day date NOT NULL, environment text NOT NULL, tenant_id uuid NOT NULL,
user_id uuid, event_type text NOT NULL, dimension text, count bigint NOT NULL,
PRIMARY KEY (day, environment, tenant_id, user_id, event_type, dimension)
);For app switches dimension holds e.g. chord:G J:closehub→journal.
Where usage events live
Written to the tenant database, keeping data tenant-resident and consistent with the isolation model everything else follows. The mgmt portal aggregates across tenants via tenant.connectionstring for operator dashboards.
If fan-out becomes slow past a few dozen tenants, a nightly job pushes usage_daily rows into the mgmt DB. That is an optimisation to apply when measured, not upfront — and it does not change the write path.
Domain metrics (business-process health)
The signals above cover whether the platform is healthy. They do not cover whether the financial processes running on it are healthy — a failed reconciliation, a close task blocked on a broken automation step, or a bank file that never arrived. Those are the failures a tenant notices first, and today an operator learns about them from a support ticket.
These are ordinary OTel metrics. No separate pipeline, no bespoke evaluator: the same SDK, collector, Prometheus and Grafana alert rules that carry http.server.duration carry these too. Only the instrumentation points differ.
The alert / diagnose split
Domain metrics tempt you into breaking the cardinality rule, because the instinct is "alert me on this failed reconciliation" — which pushes account_id, job_id, invoice_id into labels. Unbounded, and exactly the mistake that takes the backend down.
The division of labour that already exists in this design resolves it:
| Question | Where it is answered |
|---|---|
| "recon failures are up" | Metric — bounded labels, alerts fire on it |
| "which recon, for whom, and why" | Trace, log, or the jobqueue row — drill-through |
The metric fires the alert; Tempo or the row's errorlog answers the diagnosis. Nothing new is required to make this work — it is the three-pillar model applied to a new signal class.
Instrumentation points
Write paths are already funnelled, so this is a small number of edits rather than a sweep. The pattern is the same one Phase 2 uses for createModel(): instrument the chokepoint, not the call sites.
| Signal | Chokepoint | Notes |
|---|---|---|
finops.job.terminal | the statusUpdate helper wrapping UpdateObjectsActivity in the recon/journal workflows | one helper per workflow, ~25 call sites behind them; jobqueue is written on every workflow launch across finance |
finops.task.state | RefreshTaskState (refreshtaskstateactivities.ts) | already the single path that recomputes closehub.taskagentsummary |
temporal.workflow.duration | Phase 4 worker instrumentation | workflow-level failure comes free with the existing rollout |
Status values are free text, not enums. Every finance status column is VarChar(255) with the allowed values only in a doc comment, and the existing terminal check (isTerminalJobStatus) does s === "completed" || s === "failed" || s.includes("error") on a lowercased string. Any metric must normalise identically, or the counter and the database will quietly disagree.
Tables that look right and are not
Three schema objects invite instrumentation and have no writers anywhere in the application:
finbase.applicationexception— full REST CRUD, only ever reachable via its own APIfinbase.schedulerunlog— hasworkflowidanderrorlog, nothing writes it/api/jobsand its service queryprisma.job— nomodel jobexists in any schema
Instrument jobqueue and taskagentsummary, which are genuinely written. Do not build on the three above without adding the write path first.
Absence of signal
A counter only moves when something happens. If a bank file never arrives, or a scheduled job never runs, no code executes, no metric moves, and the dashboard looks healthy. This is the most common silent failure in financial operations and the one gap OTel does not close by instrumentation alone.
Two mechanisms, both standard:
- Freshness gauge — a scheduled emitter publishes
finops.feed.age_seconds{tenant, feed}; alert when it exceeds the feed's expected interval. Bounded labels. absent_over_time()in the Grafana alert rule — built into Prometheus, no new code, but only works for signals that would otherwise be emitted regularly.
The freshness gauge needs something to run on a schedule. That emitter is the only bespoke component this whole section adds; detection and delivery remain Grafana's job.
Where the emitter runs
Gauges are per-tenant, so the emitter must reach every tenant database — the same fan-out question usage events defer. Settle it once for both. A Temporal scheduled workflow fits the existing stack (durable, retries free, already how scheduled work runs here); apimgmt fan-out over tenant.connectionstring is simpler but puts a growing connection count on the request path.
Counters do not have this problem — they are emitted in-process at the moment of the write, by whichever service already holds the tenant context.
Retention
| Store | Retention | Rationale |
|---|---|---|
| Metrics | 13 months | Year-over-year comparison |
| Traces | 7–14 days | Debugging window; high volume |
| Logs | 30 days | Incident investigation |
usage_event (raw) | 12 months, then drop | Decided; DPA-covered |
usage_daily (rollups) | 24 months | Small; long trends |
Enforced by a scheduled job from day one. Retention added later is always a painful migration.
Dashboards — build vs. buy
Do not rebuild in mgmtui what Grafana already does. Dashboards-as-config, alerting, anomaly detection and ad-hoc querying are months of Vue work to end up with something worse.
| Surface | Where | Why |
|---|---|---|
| System health, latency, errors | Grafana | Alerting + ad-hoc query included |
| Business-process health (failed recon, stuck close tasks, stale feeds) | Grafana | Same alerting engine; domain metrics are ordinary metrics |
| LLM cost & token trends | Grafana | Time-series native |
| Traces / debugging | Grafana (Tempo) | Nothing custom competes |
| Tenant CRUD, onboarding | mgmtui | Operational workflow, not analytics |
| Per-user usage analytics | mgmtui | High-cardinality + joins to user/tenant names — not a Prometheus workload |
The planned "system health" views in mgmtui should be Grafana panels embedded via iframe, not custom charts. Operators keep one portal; we maintain no charting code.
Per-user shortcut analytics is the genuine exception: it needs joins to names and is queried from Postgres by apimgmt, so it stays a real mgmtui page.
The mgmtui health dashboard — re-scoped
An earlier plan for the mgmt portal's landing page was a custom health console: bespoke Vue tiles, a scheduled evaluator querying tenant databases, a findings table, and its own notification channel. That is superseded by this design. It would have rebuilt Prometheus alerting, Alertmanager and Grafana inside an application codebase — and would have needed rewriting again for the AWS move, since it was shaped around Railway specifics.
What the mgmt portal's health surface should be instead:
| Need | Implementation |
|---|---|
| System health, latency, error rates | Embedded Grafana panel |
| Business-process health (failed recon, stuck close tasks, stale feeds) | Embedded Grafana panel, fed by finops.* |
| Alerting and delivery | Grafana alert rules — not an in-app evaluator |
| Drill-through to a specific failure | Link out to Tempo, or to the existing tenant/job views |
| Tenant CRUD and onboarding | Stays a real mgmtui page — operational workflow, not analytics |
So the landing page becomes a thin composition of embedded panels plus the operational workflows that genuinely belong in the portal. No charting code, no alert-evaluation code, no findings table.
Embedding decisions still open (worth settling before the first panel ships): whether operators get Grafana logins or panels are shared via service token / anonymous-view, and how that interacts with the admins-only Cloudflare Access policy already in front of mgmt.finaisse.com. Iframe embedding also needs the portal's CSP frame-src to allow the Grafana origin.
One thing the custom approach was right about and this does not remove: the domain-level signals it was going to poll for. Those are real, and they are now finops.* metrics instead of DB polling.
Privacy
- Usage events carry user IDs and are covered by customer DPAs (confirmed).
- Never put PII in metric labels or trace attributes — those stores are not built for deletion.
- Deletion requests are satisfiable because per-user data exists only in
usage_event/usage_daily, both keyed bytenant_id+user_id. - Keep raw-event retention at 12 months; rollups carry no keystroke-level timing.
Delivery phases
Phase 1 — foundation. OTel Collector deployed (Railway now, AWS later); Grafana Cloud account and exporters; shared @blitz/telemetry lib wrapping the SDK with resource attributes; one pilot service (blitz-api) fully instrumented. Exit criterion: HTTP metrics and traces visible in Grafana, staging and production distinguishable.
Phase 2 — LLM tokens. Implement the existing design against this framework: TokenUsageHandler on the createModel() factory (catches all 29 call sites — per-call-site reads miss most usage because withStructuredOutput discards usage_metadata), emitting llm.tokens. Delivers the cost dashboard that has been wanted since July.
Phase 3 — usage events. usage_event + usage_daily tables and the write path; app-switch tracking from blitz-ui (single instrumentation point: handleModuleClick, which chord, in-menu and click all already route through); mgmtui usage dashboard.
Phase 4 — rollout. Remaining Bun services; then bapiproxy/excelrw (.NET — OTel has a mature .NET SDK) and classicml (Python). Pino → Loki with trace correlation. Alerting rules.
Phase 5 — domain metrics (blitz#1229). finops.* signals: counters at the jobqueue and taskagentsummary chokepoints, a scheduled emitter for feed freshness, and Grafana alert rules on top. Turns "a tenant reports it" into "we were alerted first". Depends on Phase 1; independent of 2–4, though it pairs naturally with Phase 4's Temporal worker instrumentation.
Phases 1–3 are independently useful; nothing later blocks earlier value.
Open questions
- Frontend telemetry. Browser-side OTel (page load, route timing, JS errors) is valuable but a separate decision — it adds bundle weight and a public-facing collector endpoint.
- Sampling. Traces at 100% are fine at current volume; head-based sampling will be needed as traffic grows. Decide before it hurts. Note this interacts with domain metrics: if traces are the drill-through from a
finops.*alert, the policy must not drop error traces — which argues for tail-based or error-biased sampling rather than plain head-based. - Cost ceiling. Grafana Cloud's free tier is generous but finite. Set an alert on approaching limits so the first surprise is not a bill.
- Grafana Cloud account ownership. Four sub-decisions, all easier to make now than to unwind once dashboards and alert rules exist in the wrong org: (a) whose account — a shared/service identity rather than a personal email, so access is not bound to one person; (b) who owns billing and sets the usage-alert threshold; (c) where the write API keys live (Railway service variables via IaC, GitHub Actions secrets, or both — never committed); (d) who gets console access and at what level, given editing alert rules is a different privilege from viewing panels. (d) also interacts with the embedding decisions.
References
- LLM Token-Usage Capture — the Phase 2 detail; its factory-instrumentation insight stands
- Infrastructure — Railway today, AWS planned
- App Switcher Shortcuts — the Phase 3 client instrumentation point