Data plane v2: implementation progress

Overview and design: Cells: 75,000 agents in parallel. This page is the detailed step log.

Updated 2026-11-17, 10:55 am CT. Step 11 is under way: session files (11.1), MCP servers with their OAuth tokens (11.2), sandboxes (11.3), schedules, triggers, long tool calls and subagents (11.4), and Slack and channels, observers, trace exports and an opt-in warehouse export (11.5) run on v2. Reports are deprecated and get nothing from v2. Next: one test org fully on v2 next to production (11.6). Step 11's goal: a full org working end to end on v2, then 75 000 agents that use files, MCP, sandboxes, schedules and subagents. That result decides between full v2, keeping v1, or a hybrid. Step 10 is done: 72 000 agents on EKS with a cell node and a quarter of the executors killed mid-run finish with 0 errors at ~200 turns/s, and the killed node now rejoins in about 3 minutes without running out of memory (run 13). The overload run passed too (run 12: 0 errors, no out-of-memory kills, refusals stopped once arrivals did). Bucket protection and a cross-region restore drill move to step 11, before the first production v2 org. Step 9 is merged (#5375): a cell that loses every node comes back from the object store, as of its newest state or a moment before. In the full-size drill (75 000 sessions, 37 534 in the lost cell), the rebuild took 3.7 s (budget 10 min), shipping lag p99 was at most 1.5 s (budget 2 s), and no tool ran twice.** Step 8 is merged (#5364): a v2 org's chat runs on its data plane end to end, behind the same URL the console and the CLI already use. A live reply chunk reaches the browser with p99 69 ms (budget 100 ms), and the edge adds p99 under 1 ms (budget 5 ms). Verified in a real browser (streaming, Stop, approvals), with the CLI, and with a real model.** Step 7 is merged (#5361): the server is now the control plane for v2 orgs. An edit reaches the data plane with p99 9.4 ms (budget 1 s), the feed arrives with p99 199 ms (budget 2 s), and allowances never overspent under concurrent retried requests.** Step 6 is merged (#5356): one cell with separate executors, the control plane protocol against a fake control plane, and the failure suite all pass; a chat turn costs the cell's leader 2.75 ms of CPU and waits for 5 round trips, both within budget.** Step 5 is merged; step 4's live check passed in prod after today's release. Design: knowledge/execution/data-plane-v2/design.md. Every step has a bench gate; a step is done when its gate holds.

Plan

#StepStatus
0Bench first, core protocols on paper✅ Done (#5282)
1Engine hooks✅ Done (#5283)
2Trusted host extensions✅ Done (#5294)
3Agent definitions as data✅ Done (#5296)
4Workers on the Engine (PostgreSQL hosting)✅ In prod (#5320); live check passed after the Nov 11 release; some features still worker-host only
5crates/cell + testkit (port funout, TLA+ models, simulation)✅ Done (#5341)
6One thin, complete cell deployment✅ Done (#5356)
7Control-plane contract✅ Done (#5361)
8Gateway, edge router, live streaming, local full v2✅ Done (#5364)
9Disaster recovery✅ Done (#5375)
10Prove on EKS (75k agents) + operations✅ Done: 72k chaos, overload and rejoin all clean on EKS; bucket protection moved to step 11
11Full org on v2: every feature on cells, one org end to end, 75k agents with the feature mix, then a decision brief (full v2, keep v1, or hybrid)🔵 In progress: 11.1 to 11.5 merged, 11.6 next
12Finalization after the decision: moving orgs to v2, v2 as the default, self-hosted upgrade, production rollout, metrics and alerts, documentation, v1 retirement⚪ After the decision

Step 11: A full org on v2

Goal (Nov 17): a full org working end to end on v2, and 75 000 agents using its features. With both in hand comes the decision: everything on v2, keep v1, or a hybrid. Step 12 is the finalization after that decision: moving orgs to v2 with a per-org move command and rollback window, v2 as the default for new orgs, the self-hosted upgrade (Helm, Docker Compose, upgrade guide), production rollout with bucket protection and a cross-region drill, dashboards and alerts with runbooks, public documentation, and retiring v1. Plan approved Nov 17. One PR per sub-step, each with its own bench line.

#Sub-stepGateStatus
11.1Session files: /fs routes, uploads, downloads, starter files, chunk cacheUpload/download p99 < 200 ms up to 1 MB; ≤ 1 commit per turn; no lost write under chaos✅ merged everruns#5461 (Nov 17): upload p99 40–53 ms, download p99 25–44 ms up to 1 MiB; 22 369 uploads under leader and follower kill -9, none lost
11.2MCP servers and per-session overrides, OAuth tokens refreshed by the data planeMCP call overhead p99 < 20 ms; a refreshed token used across executors✅ merged everruns#5465 (Nov 17): credentials opened 3.6 / 4.1 ms p50 / p99, kept 7 / 21 µs; 4 executors on one expired grant refresh it once; a person's own grants and session overrides go with 11.6
11.3Sandboxes: records in cell state, workspace sync, egress through the proxyStart within 10% of v1; unchanged sync < 50 ms; no orphaned sandbox✅ merged everruns#5466 (Nov 17): start +6.2% of provider time vs v1; unchanged 1000-file sync 1.3 ms before a call, 17.9 ms after; an orphan from a killed executor is removed before the next create
11.4Schedules and triggers, detached tool calls, agents using agents10 000 schedules fire within 1 s, also across a leader change; detached calls survive an executor kill; a subagent on another cell answers its parent✅ merged everruns#5467 (Nov 17): 10 000 schedules fire at most 121 ms late; a crash-driven leader change costs ~5 s of election (openraft has no leader hand-over yet; goes to the decision brief); a detached call resumes after its executor is killed; a subagent on the other cell answers inline and in the background; agent triggers admit their runs on v2
11.5Integrations: Slack and channels, observers and evals, trace and analytics exportSlack reply p99 < 5 s; trace export < 2 min; observers see each turn once✅ merged everruns#5469 (Nov 17): channel reply p50 334 ms, p99 721 ms; a tapped event reaches the control plane p99 468 ms; all 400 turns seen exactly once across a leader kill. Reports deprecated (no v2 data); analytics through an opt-in per-org warehouse export (hourly gzipped NDJSON in object storage). Attachments and reactions go with 11.6
11.6One test org fully on a v2 data plane next to production: console, CLI, Slack, real modelsEvery page and command it uses works, with no 501 or hidden feature⚪
11.775 000 agents with the feature mix (files, MCP, schedules, subagents, some sandboxes)~72 000 agents with chaos: 0 errors, no out-of-memory kills, cost and CPU recorded⚪

Step 10: Prove on EKS

A full fleet on the production cluster, isolated from the app: 3 cells × 3 nodes, 8 executors and 3 drivers on one 48 vCPU machine in its own node pool (back to zero after each run), simulated models, a separate bucket. No Everruns org, user or key is involved.

Runs

RunDateTurnsErrorsTurns/sWhat it found
6Nov 15723 1100199Baseline; the killed node ran out of memory 5 times while catching up
8Nov 15Saturated host: cells stopped committing ~20 min, 3 nodes out of memory. Fixed by Raft on its own listener and shedding a stalled leader (#5440)
9Nov 16655 2006 790123A killed follower never caught up (2 MB request limit on snapshots). Fixed: limit lifted, a far-behind follower steps aside (#5447)
10Nov 16409 90031 39969Rejoin fixed. New: S3 uploads went through a small NAT that ran out of burst; shipping fell behind and every cell stopped taking work. Fixed: S3 gateway endpoint (saas, Nov 16 release) and only leaders upload snapshots (#5448)
11Nov 16720 7700207.6Clean. Worst shipping lag 2.7 s (bound 5 s), no S3 timeouts. Killed node back within ~10 min, but ran out of memory 5 times on the way. Fixed: the commit index is kept on disk, so a restarted node replays its log in small chunks (#5455; local repro 2.9 → 2.1 GB, below the leader)
13Nov 16720 9900202.4Chaos with the rejoin fix: no out-of-memory kills. The killed node was back ~70 s after the kill and caught up 2 min later at 2.7 GB (run 11: killed 5 times, ~10 min). First CPU and memory record: cells 12.5 cores average (18.4 peak), executors 18.5 (27.2), drivers 0.6; the machine 69% average, 96% peak, 118 of 189 GB at peak
12Nov 16716 7000189.6Overload: bursts of 30 000 new agents, 2 KB messages, up to 46 563 inputs queued. No restarts, no out-of-memory kills; 1 893 requests refused for memory, all retried and done; no refusals after the last burst while the queue drained to 0. Turn p99 580 s (queueing)

Gate

Step 9: Disaster recovery

A cell that loses every node at once (two zones, correlated disks) is rebuilt from its last snapshot in the object store plus the log its leader shipped since. One org can be taken back to a moment without touching any other org. One PR: #5375.

Gate

Measured with the full-size drill: 2 cells on one machine, 75 000 sessions, every node of one cell killed with kill -9 while 3 000 sessions were mid-turn.

Work items

Step 8: v2 sessions end to end

A v2 org's sessions run on its data plane, behind the URL the console and the CLI already use. The control plane keeps every session's record, and the cells run the turns. Stage 1 is chat: files, MCP and sandboxes are hidden for a v2 org. One PR: #5364.

Gate

Work items

Step 7: Control-plane contract

The real server serves the control plane side of /dp/v1 for orgs on a v2 data plane, and the cells serve the rest. Orgs with no data plane (all of them today) are unchanged. One PR: #5361.

Gate

Work items

Step 6: One thin, complete cell

A cell as deployed: three state nodes, executors as separate processes that pull turns with credits, a durable workspace per session, and the control plane protocol (/dp/v1) against a fake control plane. One PR: #5356.

Gate

Work items

Step 5: Cells

funout's cell ported onto the Engine (crates/cell, unpublished): own Raft log, sessions placed per cell, live moves, spend allowances, log shipping to object storage. One PR: #5341.

Gate

Work items

Step 4: Workers on the Engine

Platform agents compile to one id-free declaration the Engine runs; no platform id enters the Framework model. A session runs on the Engine when its organization has the engine_hosting switch on and its agent compiles; everything else stays on the worker host.

Architecture review of steps 1 to 3

A review of the merged code for soundness, interfaces and dependencies. Nothing reviewed runs in production yet. Corrections: #5303.

Step 3: Agent definitions as data

Agents can be stored as data the session authority keeps, one immutable revision per edit. The Engine builds the agent locally and caches it; an edit applies from the session's next turn.

Step 2: Trusted host extensions

A hosting profile can hand an Engine turn the platform services a worker turn already uses (budgets, payment, credentials, stores, hosted MCP, Knowledge context, tool detaching). An empty bundle changes nothing.

Step 1: Engine hooks

Gate

Work items

Step 1 numbers

MeasureBeforeAfter
Authority writes per chat turn229 (5 waited)
Turns/s, 10th tenth of a long run5.4~1 000 (flat)
Engine memory per idle session6.2 KB1.3 KB
index.mdDownload .md