Telemetry here means the operator’s view: is the process healthy, is it losing data, and what happened on a request. It is deliberately separate from the tenant-facing audit log and usage surfaces, which answer product questions rather than operational ones.

​
Gateway logging

apps/gateway/crates/obol-gateway/src/telemetry.rs installs a JSON tracing subscriber filtered by RUST_LOG, defaulting to info. Initialization is idempotent. Every request produces exactly one access-log line on the target obol.access. A RequestLog is created by middleware, absorbs handler-supplied context, and emits on Drop — so a panicking or cancelled request still logs.
The access log carries ids and counts only. It never carries headers, request or response bodies, tool arguments, or any credential. A change that adds header or body logging to this path is security-sensitive and should be reviewed as such.

​
Distributed tracing

Set OTEL_EXPORTER_OTLP_ENDPOINT to a non-empty value and the gateway adds an OTLP span exporter over tonic with a batch processor, registered as the tracer obol-gateway. The value is trimmed, and an empty or whitespace-only string means “no collector” rather than an empty URI — Docker Compose always sets the variable, defaulting it to the empty string, so this distinction matters in practice.

​
Metrics

GET /metrics returns Prometheus text exposition (text/plain; version=0.0.4).
obol_llm_tokens_total and obol_cost_micros_total are incremented only on non-streaming JSON model responses (and the cost counter only when a price is configured). Streaming responses are excluded, so neither metric is a substitute for the usage projection. Use usage queries for anything that has to add up.
Two of these deserve standing alerts. obol_meter_dropped_total counts usage events the gateway could not enqueue or emit — a full 4096-slot queue, or two consecutive emit failures. This is the one usage-loss mode that happens before the Redis transport, so control’s loss latch cannot see it (Metering). Any sustained increase means billing-grade data is being lost silently. The counter is reconciled from the meter’s internal count on each /metrics scrape, so scrape it regularly. obol_snapshot_loaded dropping toward zero means the gateway is losing its compiled policy and workspace state. It is also the second readiness condition below.

​
Health endpoints

​
Gateway

The three readiness reasons are redis unreachable, no policy snapshot loaded, and draining. /startupz reports only redis unreachable and draining; a zero-snapshot install is a successful start, not a failed one. This is ADR-0006 made operational: the gateway degrades gracefully from cache during a control or Postgres blip, but it refuses to declare itself ready for customer traffic without Redis and a snapshot. Which probe a deploy target uses is the other half of that distinction: /healthz, /readyz, /startupz, and /metrics are served on the main listener (OBOL_LISTEN_ADDR, default 0.0.0.0:8080). If OBOL_METRICS_ADDR is non-empty (default 0.0.0.0:9091), a second listener additionally serves /metrics and /healthz. A metrics listener that fails to bind logs a warning and the gateway continues serving.
/metrics on the main listener has no authentication layer. Do not publish port 8080’s /metrics to the internet — terminate it at your proxy, or scrape the separate metrics port instead. Docker Compose publishes only 8080 and leaves 9091 unpublished.
The container image is distroless and has no curl, so the binary carries its own probe: obol-gateway healthcheck <url> exits 0 only on an HTTP 200 within two seconds. Compose uses it against /startupz.

​
Control

Control exposes GET /healthz, returning {"ok": true}. There is no separate readiness endpoint; schema readiness is handled by ordering instead — the Compose control and control-worker services wait for the one-shot migrate service (alembic upgrade head) to complete successfully before they start. Control’s internal API is service-authenticated with a service JWT, not open telemetry, but its failure mode is operationally relevant:
  • GET /internal/v1/workspaces/{workspace_id}/snapshot returns 503 snapshot_unavailable when the committed workspace and policy pair cannot be rebuilt. Shedding is the point: answering 200 with a body the gateway cannot load would take the tenant dark, and the gateway keeps serving its pinned snapshot and retries.
  • GET /internal/v1/keys/{key_hash} is the cache-miss path for key resolution.
A sustained rate of snapshot_unavailable means gateways are pinned to increasingly stale snapshots — worth alerting on from control’s own logs.

​
Worker cadences

The arq worker (apps/control/app/workers.py) is where every off-request-path job runs. Knowing the cadences tells you how stale each read model can be: The cadences are staggered on purpose, and refresh_expiring_credentials is the one job with run_at_startup=False — a rolling restart must not stampede every vendor’s token endpoint at once. It is also unique=True with a deployment-shared jitter second, so two replicas schedule the same job id rather than double-refreshing a rotating refresh token. If the worker is down, the gateway keeps serving. What stops is every read model: usage stops advancing, receipts stop appearing, credentials stop refreshing (and eventually expire), and snapshot publications stop retrying. Worker liveness is not optional for long.

​
Draining on deploy

apps/gateway/crates/obol-gateway/src/drain.rs runs on SIGTERM or Ctrl-C. The sequence, bounded by one total deadline:
1

Fail readiness first

readyz and startupz start answering 503 with reason draining, so the load balancer stops sending new traffic while the process is still serving what it has. /healthz stays 200.
2

Stop accepting, finish in flight

The shutdown watch fires; axum::serve stops accepting connections and awaits in-flight ones. Background loops wind down.
3

Settle reservations

Meter::wait_for_reconciliations waits for every outstanding spend reconciliation, so a shutdown does not leave pessimistic reservations charged.
4

Drain the meter queue

The meter’s shutdown signal fires last and the flusher drains whatever is still queued to stream:usage before exiting.
If the whole sequence exceeds OBOL_DRAIN_TIMEOUT_S (default 30), a drain timed out warning is logged and every remaining task is aborted — which can drop queued usage events and increment obol_meter_dropped_total.
The default drain timeout (30s) is shorter than the default upstream timeout (OBOL_UPSTREAM_TIMEOUT_S, 120s). A long-running streaming request in flight at shutdown can therefore be aborted by the drain deadline. Raise OBOL_DRAIN_TIMEOUT_S toward your upstream timeout if you want deploys to wait out the longest in-flight stream, and give your orchestrator a matching termination grace period.
Configuration for both timeouts is validated at startup against the idempotency TTL: OBOL_IDEMPOTENCY_TTL_S must exceed the combined upstream and drain timeouts, and the process refuses to start otherwise.

Gateway

What the process these signals describe actually does.

Metering

Where dropped meter events and ingestion gaps come from.

Docker Compose

The environment variables named on this page, in context.