---
title: "Telemetry"
description: "Operational signals for running Obol: the gateway's JSON access log, Prometheus metrics, optional OTLP tracing, health and readiness endpoints on both planes, and what draining does on a deploy."
---
Telemetry here means the operator's view: is the process healthy, is it losing data, and what happened on a request. It is deliberately separate from the tenant-facing [audit log](/observability/audit-log) and [usage](/billing/usage) surfaces, which answer product questions rather than operational ones.
## Gateway logging
`apps/gateway/crates/obol-gateway/src/telemetry.rs` installs a JSON `tracing` subscriber filtered by `RUST_LOG`, defaulting to `info`. Initialization is idempotent.
Every request produces exactly one access-log line on the target `obol.access`. A `RequestLog` is created by middleware, absorbs handler-supplied context, and emits on `Drop` — so a panicking or cancelled request still logs.
| Field | Notes |
|---|---|
| `request_id` | Also the `x-request-id` header on the response, including error responses |
| `route`, `method`, `status` | |
| `latency_ms` | Measured from `RequestLog` construction |
| `key_id`, `workspace_id` | Ids, never the key secret |
| `model`, `tool` | Whichever applies to the route |
| `error_code` | The closed error vocabulary, empty on success |
The access log carries ids and counts only. It never carries headers, request or response bodies, tool arguments, or any credential. A change that adds header or body logging to this path is security-sensitive and should be reviewed as such.
### Distributed tracing
Set `OTEL_EXPORTER_OTLP_ENDPOINT` to a non-empty value and the gateway adds an OTLP span exporter over tonic with a batch processor, registered as the tracer `obol-gateway`. The value is trimmed, and an empty or whitespace-only string means "no collector" rather than an empty URI — Docker Compose always sets the variable, defaulting it to the empty string, so this distinction matters in practice.
## Metrics
`GET /metrics` returns Prometheus text exposition (`text/plain; version=0.0.4`).
| Metric | Type | Labels | Watch it for |
|---|---|---|---|
| `obol_requests_total` | counter | `route`, `status` | Traffic and error rate |
| `obol_llm_tokens_total` | counter | `model`, `dir` | Token throughput by requested model |
| `obol_cost_micros_total` | counter | — | Estimated vendor cost in micro-USD |
| `obol_mcp_requests_total` | counter | `method`, `status` | MCP method mix and failures |
| `obol_policy_decisions_total` | counter | `action`, `decision` | Allow / deny / approval-required rates |
| `obol_meter_dropped_total` | counter | — | **Usage events lost before the stream** |
| `obol_invocation_finalize_failures_total` | counter | — | Receipts that could not be persisted |
| `obol_snapshot_loaded` | gauge | — | Workspaces with a loaded policy snapshot |
`obol_llm_tokens_total` and `obol_cost_micros_total` are incremented only on non-streaming JSON model responses (and the cost counter only when a price is configured). Streaming responses are excluded, so neither metric is a substitute for the usage projection. Use [usage queries](/billing/usage) for anything that has to add up.
Two of these deserve standing alerts.
`obol_meter_dropped_total` counts usage events the gateway could not enqueue or emit — a full 4096-slot queue, or two consecutive emit failures. This is the one usage-loss mode that happens *before* the Redis transport, so control's loss latch cannot see it ([Metering](/billing/metering)). Any sustained increase means billing-grade data is being lost silently. The counter is reconciled from the meter's internal count on each `/metrics` scrape, so scrape it regularly.
`obol_snapshot_loaded` dropping toward zero means the gateway is losing its compiled policy and workspace state. It is also the second readiness condition below.
## Health endpoints
### Gateway
| Endpoint | Behavior |
|---|---|
| `GET /healthz` | Always `200` with the body `ok`. Liveness only |
| `GET /readyz` | `200` with `{"ready": true, "snapshots": n}` when Redis pings, at least one policy snapshot is loaded, and the process is not draining. Otherwise `503` with `{"ready": false, "reasons": [...]}` |
| `GET /startupz` | `200` with `{"startup": true, "snapshots": n}` when Redis pings and the process is not draining — empty snapshots are OK. Otherwise `503` with `{"startup": false, "reasons": [...]}` |
| `GET /metrics` | Prometheus exposition |
The three readiness reasons are `redis unreachable`, `no policy snapshot loaded`, and `draining`. `/startupz` reports only `redis unreachable` and `draining`; a zero-snapshot install is a successful start, not a failed one. This is [ADR-0006](/concepts/architecture) made operational: the gateway degrades gracefully from cache during a control or Postgres blip, but it refuses to declare itself ready for customer traffic without Redis and a snapshot.
Which probe a deploy target uses is the other half of that distinction:
| Target | Probe | Why |
|---|---|---|
| Compose healthcheck | `GET /startupz` | One check, feeding `depends_on` and `up --wait`. A first-boot empty install must become healthy; Redis gone must not. |
| Helm / Fly liveness | `GET /healthz` | Process is serving. Redis down is not a reason to restart the binary. |
| Helm readiness / Fly `http_service.checks` | `GET /readyz` | Customer traffic. Hosted and k8s installs have a published snapshot before they take traffic; a snapshot-less replica stays out of the Service. Do not switch this to `/startupz`. |
`/healthz`, `/readyz`, `/startupz`, and `/metrics` are served on the main listener (`OBOL_LISTEN_ADDR`, default `0.0.0.0:8080`). If `OBOL_METRICS_ADDR` is non-empty (default `0.0.0.0:9091`), a second listener additionally serves `/metrics` and `/healthz`. A metrics listener that fails to bind logs a warning and the gateway continues serving.
`/metrics` on the main listener has no authentication layer. Do not publish port 8080's `/metrics` to the internet — terminate it at your proxy, or scrape the separate metrics port instead. Docker Compose publishes only `8080` and leaves `9091` unpublished.
The container image is distroless and has no `curl`, so the binary carries its own probe: `obol-gateway healthcheck ` exits `0` only on an HTTP 200 within two seconds. Compose uses it against `/startupz`.
### Control
Control exposes `GET /healthz`, returning `{"ok": true}`. There is no separate readiness endpoint; schema readiness is handled by ordering instead — the Compose `control` and `control-worker` services wait for the one-shot `migrate` service (`alembic upgrade head`) to complete successfully before they start.
Control's internal API is service-authenticated with a service JWT, not open telemetry, but its failure mode is operationally relevant:
- `GET /internal/v1/workspaces/{workspace_id}/snapshot` returns `503 snapshot_unavailable` when the committed workspace and policy pair cannot be rebuilt. Shedding is the point: answering `200` with a body the gateway cannot load would take the tenant dark, and the gateway keeps serving its pinned snapshot and retries.
- `GET /internal/v1/keys/{key_hash}` is the cache-miss path for key resolution.
A sustained rate of `snapshot_unavailable` means gateways are pinned to increasingly stale snapshots — worth alerting on from control's own logs.
## Worker cadences
The arq worker (`apps/control/app/workers.py`) is where every off-request-path job runs. Knowing the cadences tells you how stale each read model can be:
| Job | Cadence | Effect |
|---|---|---|
| `drain_usage_events` | every 5s | Usage events into Postgres |
| `deliver_pending_meter_events` | at `:02, :12, :22, :32, :42, :52` | Stripe meter delivery |
| `drain_invocation_audit` | every 10s | Receipts read model |
| `drain_webhook_evidence` | every 10s | Webhook evidence correlation |
| `reconcile_verification` | at `:03, :13, :23, :33, :43, :53` | Asynchronous effect verification |
| `retry_pending_snapshot_publications` | every 10s | Snapshot publish outbox |
| `drain_approval_deliveries` | every 10s | Approval notifications |
| `resume_approved_calls` | at `:05, :15, :25, :35, :45, :55` | Held invocations resuming |
| `refresh_expiring_credentials` | every 5 min | OAuth credential rotation |
The cadences are staggered on purpose, and `refresh_expiring_credentials` is the one job with `run_at_startup=False` — a rolling restart must not stampede every vendor's token endpoint at once. It is also `unique=True` with a deployment-shared jitter second, so two replicas schedule the same job id rather than double-refreshing a rotating refresh token.
If the worker is down, the gateway keeps serving. What stops is every read model: usage stops advancing, receipts stop appearing, credentials stop refreshing (and eventually expire), and snapshot publications stop retrying. Worker liveness is not optional for long.
## Draining on deploy
`apps/gateway/crates/obol-gateway/src/drain.rs` runs on `SIGTERM` or `Ctrl-C`. The sequence, bounded by one total deadline:
`readyz` and `startupz` start answering `503` with reason `draining`, so the load balancer stops sending new traffic while the process is still serving what it has. `/healthz` stays `200`.
The shutdown watch fires; `axum::serve` stops accepting connections and awaits in-flight ones. Background loops wind down.
`Meter::wait_for_reconciliations` waits for every outstanding spend reconciliation, so a shutdown does not leave pessimistic reservations charged.
The meter's shutdown signal fires last and the flusher drains whatever is still queued to `stream:usage` before exiting.
If the whole sequence exceeds `OBOL_DRAIN_TIMEOUT_S` (default 30), a `drain timed out` warning is logged and every remaining task is aborted — which can drop queued usage events and increment `obol_meter_dropped_total`.
The default drain timeout (30s) is shorter than the default upstream timeout (`OBOL_UPSTREAM_TIMEOUT_S`, 120s). A long-running streaming request in flight at shutdown can therefore be aborted by the drain deadline. Raise `OBOL_DRAIN_TIMEOUT_S` toward your upstream timeout if you want deploys to wait out the longest in-flight stream, and give your orchestrator a matching termination grace period.
Configuration for both timeouts is validated at startup against the idempotency TTL: `OBOL_IDEMPOTENCY_TTL_S` must exceed the combined upstream and drain timeouts, and the process refuses to start otherwise.
What the process these signals describe actually does.
Where dropped meter events and ingestion gaps come from.
The environment variables named on this page, in context.