---
title: "Sessions, snapshots, and drain"
description: "How the gateway holds MCP session state in Redis, consumes workspace and policy snapshots, and drains gracefully on shutdown."
---
The gateway runs as two or more stateless replicas. Nothing that matters lives
in a single process: MCP sessions are in Redis, workspace and policy state
arrives as snapshots, and shutdown is bounded by one deadline.
## MCP session ids
A Streamable HTTP MCP session begins at `initialize`, which returns an
`Mcp-Session-Id` header that every later request presents.
Obol's session ids are **opaque**: the literal prefix `mcs_` followed by a UUID
v7, validated on parse (ADR-0013). The id carries no state. The record lives in
Redis at `sess:{session_id}`:
| Field | Meaning |
|---|---|
| `session_id` | The `mcs_…` id |
| `workspace_id` | The tenant the session belongs to |
| `key_id` | The virtual key that created it |
| `protocol_version` | The negotiated MCP protocol version |
| `upstreams` | Informational record of upstream sessions |
| `created_at`, `last_seen` | Timestamps |
The `upstreams` field is informational. Upstream MCP sessions are held per
replica by the connection pool and rebuilt on demand, with a 45-second maximum
age so an upstream session never outlives the 60-second service JWT that
authorizes it. Correctness never depends on an upstream session surviving a
replica switch.
### Why opaque rather than self-describing
An alternative design encrypts the set of upstream session ids into the
client-facing id, which removes the shared store. Obol rejected it: rotating the
encryption key would invalidate every live session, the id length would grow
with the number of upstreams, and workspace binding would have to be layered on
top anyway. Redis is already on the hot path for authentication, so the session
lookup adds one round trip to a request that was already making one.
### Expiry
The Redis TTL is the idle timeout, `OBOL_SESSION_TTL_S` (default 1800 seconds).
Every create, save, and resume rewrites the record and refreshes the TTL. There
is no reaper task. Creation sets the key with the TTL; renewal is a conditional
compare-and-set that never recreates a record that has already expired or been
deleted.
`DELETE /mcp` removes the session and returns `204`. Revoking a key can also
delete its sessions.
### Workspace binding
`get_or_resume(id, workspace)` returns "no such session" for three distinct
cases: a missing id, a malformed id, and a record whose `workspace_id` differs
from the caller's. All three render as `404`, so a cross-workspace probe is
indistinguishable from a typo. The cross-workspace case is logged as a warning
carrying the workspace id and the session's last four characters only.
A session store error is `503`, never a session-less fallthrough.
A session id is a bearer credential for its lifetime. It is never logged in full
— only `session_last4` — and a test asserts that no log line contains a session
id or an authorization secret.
## Snapshot consumption
The gateway holds no Postgres connection on the request path. Control is the
sole writer of Postgres; it compiles per-workspace state and publishes it to
Redis, and the gateway hot-reloads (ADR-0006).
Two documents make up a publication:
- **Workspace snapshot** at `snap:workspace:{workspace_id}` — connections,
tools, models, prices, encrypted credentials, environment, frozen flag.
- **Policy snapshot** at `policy:{workspace_id}` — the compiled Cedar bundle.
They are read and pinned together. A request calls `pair` once and uses that one
publication for routing *and* authorization, so a snapshot swap mid-request
cannot route against one revision and authorize against another. Every fallback
model in an LLM request is checked under the same pinned publication as the
primary.
### The read path
A cached pair less than 30 seconds old is returned directly.
On a miss or after 30 seconds, one `MGET` reads the workspace snapshot, the
policy snapshot, and their shared fingerprint. Only a validated, coherent pair
is installed. A failed revalidation does **not** extend the previous pair's
freshness — expired reads fail closed. A missing fingerprint or a partial pair
is an error, not a reason to fall back.
On a complete Redis miss, the gateway asks control's internal snapshot API with
a service JWT. An atomic operation creates all three publication keys only if
none exists, publishes an invalidation, and the gateway reads back the
authoritative winning pair. A delayed response never overwrites an existing
publication.
At startup, `run` preloads up to 1000 workspaces by scanning for policy keys.
`/readyz` requires at least one loaded snapshot plus a successful Redis ping.
### Invalidation
Control publishes tagged `Invalidation` messages on the Redis `obol:invalidate`
channel. The gateway runs a dedicated subscriber connection that reconnects with
backoff from 100 ms up to 5 s.
| Invalidation | Effect |
|---|---|
| `Workspace` / `Policy` | Re-read both snapshots for that workspace; drop from memory if the key is gone |
| `Key` | Handled by the auth layer, which purges `key:{hash}` |
| `All` | Reconcile every cached workspace, then preload |
Messages are rebroadcast to downstream listeners even when the reload itself
fails, so stale state is dropped either way. Malformed payloads are logged and
discarded. Immediately after subscribing or reconnecting, and every 30 seconds
after that, known workspace ids are reconciled to repair missed notifications.
TTLs remain the fallback when pub/sub is unavailable: a cached `KeyContext`
expires after `OBOL_KEY_TTL_S` (default 60 seconds) regardless.
`POST /internal/reload` lets control force a reload without waiting for the
channel.
## Drain
Shutdown is triggered by `SIGTERM` or `SIGINT` and bounded by one
`OBOL_DRAIN_TIMEOUT_S` deadline (default 30 seconds) covering the whole
sequence:
Readiness flips to draining, so `/readyz` returns `503` with reason `draining`
and the load balancer stops sending new connections. `/healthz` still returns
`200` — the process is alive, just not accepting work.
The HTTP servers stop accepting and wait for in-flight connections to finish.
Background tasks — the invalidation subscriber, the metering flusher's feeders —
are signalled through the same shutdown channel.
The meter waits for outstanding reservation reconciliations, so a request that
reserved budget settles it rather than leaving an over-reservation behind.
The meter's own shutdown flag flips, the flusher drains its queue, and exits.
If the deadline expires before all of that completes, remaining tasks are
aborted and a warning is logged with the timeout value. Because a usage guard
emits on drop — including on cancellation — an aborted request still produces a
usage event marked `partial`.
The idempotency TTL is validated at startup to exceed the upstream plus drain
timeouts, so an invocation cannot outlive its own record during a drain.
## Related
What runs inside a session: namespacing, filtering, invocation.
Where snapshots come from and who writes them.
Why Postgres is never on the streaming path.
What the flusher writes and how usage settles.