Metering is not the same thing as Obol’s invoice. The cost figures in a usage event are your estimated vendor cost from your own pricebook. The design-partner offer charges per startup plus logical dispatched tool/API and LLM requests, on the grandfathered 3 / 1 per 1,000 meters — not the public 1.00 / 0.10 ladder. Both schedules are named on Stripe Billing.
What is metered
A usage event has one of two kinds, frompackages/proto/usage_event.schema.json:
Every event carries
workspace_id, key_id, request_id, kind, usd_micros, pricing_status, latency_ms, HTTP status, and a partial flag; llm events add model, provider, and the token block; tool events add tool, tool_calls, and the receipt_id when the call produced a receipt.
Token counts are normalized to the “input excludes cached” convention: tokens_in never double-counts tokens_cached, and tokens_reasoning is informational only — providers already fold it into tokens_out, so it is never billed twice.
Priced versus unpriced
pricing_status is explicit, because a numeric zero is ambiguous. known means the workspace pricebook had a rate for this model or tool; unknown means it did not, and the usd_micros zero is a sentinel rather than a measurement. Aggregates keep the two apart (usage queries report a known subtotal plus an unpriced_count).
In the gateway: the obol-meter crate
apps/gateway/crates/obol-meter owns pricing arithmetic, admission counters, and event emission. It never touches a credential — events carry ids and counts only.
Cost
Pricing::cost works in integer micro-USD. Each term is computed in i128, rounded half-up, and the sum saturates at i64::MAX. Cached tokens use the pricebook’s cached_input_per_mtok when present and the input rate otherwise. There is no floating point anywhere on this path.
Admission and reservation
Before an upstream is opened,Limiter charges rate limits and reserves spend against Redis fixed windows. Any store error fails closed — Redis unreachable means not_ready, not “allow and hope”.
A model request reserves a pessimistic token bound and maximum cost atomically (
admit_llm), then reconciles down to the observed usage when the request finishes. A tool call reserves its configured fixed price immediately before the first upstream dispatch (reserve_tool, ADR-0046) and reserves no tokens at all. A budgeted tool call with no configured price fails closed with PricingUnavailable rather than executing for free.
Reconciliation is tracked, not fire-and-forget: the drain sequence waits for outstanding reconciliations before closing the meter queue. If reconciliation fails, the pessimistic reservation stays charged and reconciliation_failures increments — the safe direction.
The UsageGuard
One UsageGuard exists per upstream attempt and emits exactly one usage event on Drop. That covers normal completion, panic unwind, and future cancellation alike, so a client that disconnects mid-stream still produces a usage event — flagged partial: true, since the upstream never signalled completion.
Dispatch facts
AUsageDispatchEvent records the gateway-owned logical identity, first dispatch timestamp, and reviewed tool effect class. It is emitted when dispatch starts, before completion, so a long-running request does not defer ordinary billing ingestion to the next period. Completion events retain the same metadata for recovery. LLM fallbacks retain one logical identity while their token and estimated vendor cost events remain separate.
This fact uses the same asynchronous transport and its loss limitations. It is not a durable proof that the source is complete. Usage-bearing design-partner invoices require operator review even after observed totals reconcile.
The async emitter
Meter is a bounded mpsc channel feeding a batching flusher:
Enqueue is a non-blocking
try_send. When the queue is full the event is dropped, counted, and logged — a response is never delayed to make room. Emission retries once; a second failure drops the event and increments the same counter. That counter surfaces as obol_meter_dropped_total on /metrics (see Telemetry), and it is the one usage-loss mode that happens before the transport and is therefore invisible to control’s loss latch.
The transport
The flusher writes each event to the Redis streamstream:usage with XADD ... MAXLEN ~ 1000000. The entry envelope is three fields — type (usage or dispatch), workspace_id, and the JSON payload. The stream is shared infrastructure; tenant identity rides in every envelope.
Because the stream is capped, it is a bounded transport. Sustained ingest lag can trim entries that control has not yet consumed. Detecting that is the whole point of the next section.
In control: ingest
apps/control/app/services/usage_ingest.py runs inside the arq worker as drain_usage_events, scheduled every five seconds (second={0,5,10,...,55}, run_at_startup=True).
1
Sample the stream for loss
Read the consumer group’s position and compare the max deleted entry id against the last delivered id. Any gap latches immediately.
2
Reclaim stalled entries
XAUTOCLAIM on the control-usage group for entries idle longer than OBOL_USAGE_CLAIM_IDLE_MS (default 60000), resuming from an in-process scan position so old pending entries are not starved behind fresh ones.3
Read fresh entries
XREADGROUP fills the remainder of the batch (OBOL_USAGE_BATCH, default 256), using the same atomic sampling-and-latching script so a trim between claim and read cannot hide loss.4
Project and store
Each entry is validated against the frozen JSON Schema and projected into an immutable
usage_events row. Legacy unenrolled tool traffic retains its usage_meter_delivery path. A design-partner dispatch fact creates a separately deduplicated immutable billing allocation; completion metadata can recover the same fact. Enrolled traffic never enters both meter paths.5
Commit, then acknowledge
The SQL commit happens before
XACK. A crash between them replays the entry, and the event id’s primary key makes the replay a no-op duplicate.Per-entry outcomes
Only the trusted outer envelope attribute decides the tenant. A quarantine row stores the stream entry id, the workspace, a reason code, and a fixed diagnostic string — never the rejected event id and never the payload, because an id-shaped field in a rejected payload could carry injected content.
Loss latching (ADR-0045)
RedisXAUTOCLAIM removes deleted pending entries from the pending list while returning their ids. If the SQL marker commit then failed, the only evidence that usage went missing would be gone. So the evidence is made durable in Redis before it is consumed.
The latch is a sentinel consumer group named control-usage:loss-detected on the existing stream:usage key. It holds only shared transport health: no tenant identifier, no payload, no credential, no receipt. A Redis script samples stream and group metadata and latches any gap before returning entries; the claim script creates the sentinel before XAUTOCLAIM’s destructive pending-list cleanup, and only a clean claim that found no loss may remove a temporary sentinel it created itself.
Once latched, the state survives every later healthy call, every SQL failure, and every worker restart. Each drain propagates it into a sticky usage_ingestion_gap_at stamp on every existing workspace, with two visible consequences:
- Usage queries report
"completeness": "gap_detected"instead of"complete". - Stripe meter delivery for that workspace pauses (
paused_gap) rather than reporting a knowingly incomplete count.
Configuration
Control worker environment
control-worker service in infra/docker-compose.yml; see Docker Compose.
What this is, and is not
This pipeline produces a bounded-source projection, not invoice-complete accounting. It is eventually consistent by construction: budgets lag by seconds, the usage read model lags by a drain cycle, and the read model tells you when it knows it is incomplete rather than presenting a gap as a total.Stripe Billing
What actually reaches Stripe, and why it is never your payment volume.
Querying usage
The usage summary endpoint, its windows, and budget posture.