---
title: "Architecture"
description: "Obol's two planes, the deployables that actually ship, and the snapshot pipeline that carries state from Postgres to the Rust hot path."
---
Obol is two planes in one monorepo. ADR-0005 named four deployables; three of them ship — `apps/gateway`, `apps/control`, and `apps/web` — alongside `apps/landing`, which the ADR predates and which only Compose builds. The fourth, `apps/connectors`, is a reserved design with no source and no service in any deployment target. The split exists because the two halves have opposite profiles: the data plane is a hot proxy where tail latency and memory matter, and the control plane is a product where iteration speed and library ecosystem matter (ADR-0002).
ADR references throughout these docs point at the decision records in `docs/adr/` in the Obol repository. Accepted ADRs are immutable — a reversal supersedes one rather than editing it.
## System shape
```text System shape
Clients (apps, Cursor, Claude, OpenAI SDK)
|
| Bearer ob_live_... / MCP OAuth JWT
v
+---------------------+
| DATA PLANE (Rust) | apps/gateway
| /v1/* and /mcp |
| auth, CEL, Cedar |
| stream, mux, vault |
+----------+----------+
| snapshots only
Redis <----+----> KMS (KEK)
|
+----------v----------+
| CONTROL (Python) | apps/control
| keys, OAuth, Cedar |
| billing, approvals |
+----------+----------+
|
Postgres + Stripe Billing
|
+----------v----------+
| WEB (Next.js) | apps/web
+---------------------+
```
The gateway's outbound edge fans out to LLM providers under the customer's own key, an in-process OpenAPI executor for generic REST, and federated remote-MCP or catalog backends. A `worker://` arm is compiled in but has nothing to dial: no worker is deployed in any target.
## The deployables
| App | Language | Ships as | Owns |
|---|---|---|---|
| `apps/gateway` | Rust | one static binary | Data plane: LLM and MCP proxy, OpenAPI executor, auth, CEL and Cedar, vault unwrap, metering emit |
| `apps/control` | Python (FastAPI) | API and worker images | Tenants, key mint, OAuth orchestration, Cedar compile and publish, approvals, Stripe Billing, audit |
| `apps/connectors` | — | nothing; no image, no deployed service | **Reserved design, not built** (parked by ADR-0058). Intended for reviewed request planners and response normalizers for messy vendors; observations only, never receipt authority |
| `apps/landing` | TypeScript (Next.js) | web app | Marketing site; built by Compose only, no backend dependency |
| `apps/web` | TypeScript (Next.js) | web app | Dashboard: keys, connectors, policy editor, receipts, approvals, usage |
### Data plane — `apps/gateway`
The gateway is one binary with two listeners. It is never split into an LLM proxy and an MCP proxy, because sessions, auth, policy, and metering have to share memory.
It listens on:
- `POST /v1/chat/completions`, `/v1/responses`, `/v1/embeddings`, `/v1/messages`
- `POST | GET | DELETE /mcp` (Streamable HTTP, `Mcp-Session-Id`)
- `POST /v1/route` for capability resolution
- `POST /hooks/{workspace}/{connection}/{subscription}` for signature-verified vendor webhooks
- `GET /healthz`, `/readyz` (Redis reachable, ≥1 policy snapshot, not draining), and `/startupz` (Redis reachable and not draining; empty snapshots OK)
- An internal surface for reload, credential exchange, refresh, seal, catalog session, verification readback and evaluation, and approval resume
It does:
`Bearer ob_live_...` resolves by SHA-256 hash against Redis, falling back to control's snapshot API; MCP OAuth 2.1 JWTs are validated on `iss`, `aud`, and scopes.
Key active, workspace not frozen, model or tool allowed, budget remaining, RPM and TPM within limits.
CEL prefilter for visibility, then Cedar `is_authorized` as the decision of record. A `forbid` on a destructive tool in production emits an approval ticket and returns a structured `approval_required` error.
Unwrap the outbound credential in-process, inject it on the outbound hop only, and route to an LLM provider, a remote or catalog MCP server, or the `openapi://` executor. A `worker://` executor is compiled in but has nothing to dial; it still speaks ADR-0014, not ADR-0024.
Normalize every executor result into a `DispatchObservation`, run the deterministic `EffectSpec` verifier, and mint an `InvocationReceipt` plus a `VerificationEvent` — committed together as one `InvocationAuditEnvelope` before any terminal result or receipt metadata is exposed.
It does **not** run OAuth consent, render UI, talk to Stripe Billing, embed CPython, or hold connector business logic.
The crate layout is deliberately thin and leaf-heavy — leaf crates depend only on `obol-types`, and composition happens in `obol-gateway` through store, executor, and hook traits (ADR-0018). See [Gateway crates](/gateway/crates) for the full map.
One structural detail worth noting here: capability discovery lives in `obol-discover`, which may depend on `obol-types` alone. The *absence* of an edge from `obol-route` is what keeps retrieval off the dispatch path, and a DAG check script enforces it (ADR-0029).
### Control plane — `apps/control`
Control owns Postgres and everything that is a product rather than a proxy: workspaces (each carrying an `org_id` column — there is no `organizations` table), memberships and invitations, virtual keys, the connector catalog and connections, OAuth orchestration, policy validation and publication, the approvals inbox, receipts, usage, and the audit log, all under `/api/v1`.
Background workers (arq) handle OAuth exchange and refresh scheduling, Cedar compile and snapshot publish, key-cache invalidation on revoke, usage and receipt flush to Postgres and Stripe meters, verification scheduling, webhook correlation, and deadline reconciliation.
ConnectorBundle validation and publication are **not** among them. Control reads a
reviewed pack straight off disk (`services/tools.py::load_bundle` →
`packages/connectors//bundle.json`); there is no `connector_bundles` table and no
publisher. Bundle *validation* exists in Rust (`obol-types/src/bundle.rs`).
Cedar validation runs through a small vendored Rust helper binary, `obol-policy-check`, invoked as a subprocess. Cedar is never reimplemented in Python, and that binary is the only Rust inside the control image. The subprocess boundary is a versioned contract: the CLI emits one `PolicyCompileReport` on stdout for both outcomes, and its stderr is never parsed and never reaches an operator response.
### Connectors — `apps/connectors` (reserved, not built)
Reviewed Python workers are designed to run out of process and be called over MCP (ADR-0007). Embedding CPython in the gateway was rejected: the GIL fights Tokio, one bad connector takes the process down, and connectors could not scale independently.
**Nothing is deployed here.** The trusted-worker tier is parked (ADR-0058): `apps/connectors/` holds one README — no package, no image — and no deployment target defines a connector-worker service. Every shipped connector runs in the gateway's in-process OpenAPI executor or federates over remote MCP.
ADR-0024 specifies that a worker returns a bounded, bundle-declared request plan, and that the gateway validates it, injects the credential, and performs the HTTP itself — so that a worker never sees a vendor credential and never calls a vendor. **That protocol is not implemented.** The compiled `WorkerExecutor` still runs ADR-0014's hand-off and would pass the vendor credential to a worker in `x-obol-upstream-authorization`. No worker exists to receive it, so no credential crosses that hop today. A worker never mints a receipt or assigns evidence trust on either path. See [Trusted workers](/connectors/trusted-workers).
### Dashboard — `apps/web`
Screens only. The browser talks to control and never to the gateway, and there is deliberately no browser-side playground: the dashboard hands over client configuration and shows the receipt that results. Human authentication is Clerk (ADR-0033); control still owns workspaces, membership, and every `workspace_id`, and Cedar remains solely the authorization engine for agent traffic.
## How state flows
Postgres is the sole source of truth and is written only by control. The gateway holds no Postgres pool on the request path (ADR-0006).
```text State flow
Postgres (truth, control-owned)
| compile: keys + connections + tools + Cedar policy set and entities
v
Snapshot publication --(one Redis transaction, read back)--> Redis
| |
| invalidation on channel obol:invalidate |
v v
Gateway snapshot cache <-- MGET workspace bytes + policy bytes + fingerprint
|
| request pins one immutable pair for its whole lifetime
v
CEL visibility -> Cedar permission -> dispatch -> receipt
|
| async: usage + invocation-audit envelopes on Redis streams
v
Control workers flush to Postgres and Stripe meters
```
Three properties matter:
- **Snapshots are the only way state reaches the data plane.** Control compiles a per-workspace bundle of keys, connections, tools, and the Cedar policy set with entities, publishes it to Redis, and the gateway hot-reloads. On a cache miss the gateway calls control's internal snapshot API with a service JWT.
- **Publication is monotonic and fingerprinted.** One Redis operation compares the workspace version in the candidate pair, refuses a lower version outright, and writes both snapshot keys, both invalidations, and the pair fingerprint together. A gateway reads the bytes and the fingerprint in one `MGET` and validates the exact-byte digest; a missing or mismatched fingerprint is not an admissible publication. Each cached pair is revalidated after 30 seconds, and once that window expires, reads fail closed until validation succeeds.
- **Writes off the hot path are async and batched.** The gateway emits usage events and invocation-audit envelopes to Redis streams; control workers drain them into Postgres and Stripe meters. A token stream never blocks on a database write.
The accepted cost is eventual consistency: budgets lag by seconds, and revoke propagation is bounded at roughly 60 seconds, which is the product's stated SLO rather than an implementation accident.
### What lives in Redis
| Key | Holds |
|---|---|
| `key:{sha256}` | `KeyContext` JSON, ~60s TTL, purged on revoke |
| `rpm:{key_id}:{yyyyMMddHH}` | Request-rate counter |
| `spend:{key_id}:{yyyyMM}` | Spend counter in micro-USD |
| `sess:{mcs_...}` | MCP `SessionRecord`, TTL equal to the idle timeout |
| `policy:{workspace_id}` | Compiled Cedar text, entities, and hash |
| `snap:workspace:{id}` | Connections, versioned connector bundles, tools |
| `idem:{workspace_id}:...` | Invocation ownership and terminal replay |
| `stream:usage` | Bounded usage event stream |
| `stream:invocation-audit:{workspace_id}` | Untrimmed receipt and verification envelopes |
| `stream:approvals` | Approval tickets, awaited on write, never gateway-trimmed |
Every Redis key carries `workspace_id`, exactly as every Postgres row does. See [Tenancy and identity](/concepts/tenancy-and-identity).
These multi-key scripts target the current single-Redis deployment. Redis Cluster would need a later ADR and a live-key migration plan (ADR-0032).
## Repo layout
```text Repository
apps/gateway Rust data plane — axum; /v1/* + /mcp; auth, CEL, Cedar,
vault unwrap, session mux, metering emit
apps/control Python control plane — FastAPI; tenants, keys, OAuth,
Cedar compile/publish, approvals, billing, audit
apps/connectors reserved for Python MCP workers — README only, nothing built
apps/web Next.js dashboard — talks only to control
apps/landing Next.js marketing site — static, no backend deps
packages/proto wire contracts (JSON schemas) — freeze before handlers
packages/cedar Cedar schema + policies + fixture tests
packages/connectors reviewed connector packs, plus the frozen catalog index
packages/fixtures synthetic connector traces; no live credentials
infra/ docker-compose (dev == Enterprise self-host), fly/, helm/
docs/ PRD, ARCHITECTURE, ADRs, GLOSSARY
```
`packages/proto` is the contract pack, and it is contract-first: freeze the schema before writing handlers, and if Rust and Python disagree on a field name, the contract pack is wrong. The JSON schemas are generated from `apps/gateway/crates/obol-types` with schemars and drift-checked in CI (ADR-0017).
## Where the boundaries hold
Two boundaries are load-bearing enough to restate:
- **No Python and no product logic in the gateway.** Generic OpenAPI and remote MCP execute in Rust. Messy first-party vendors are designed as out-of-process workers under ADR-0024; **that protocol is not implemented**, and the trusted-worker tier is parked. A feature that smells like product goes to control by default.
- **The gateway owns invocation identity, evidence classification, verification, and receipts.** Executors report observations. A worker, a remote MCP server, or a catalog response is evidence, never a receipt, and no vendor or broker payload can promote its own evidence class (ADR-0019).
The nine rules these boundaries encode.
Request paths, listeners, and dispatch in detail.
Compile, publish, fingerprint, hot-reload.
Compose, Fly, and Helm targets.