Cells: 75,000 agents in parallel
An Everruns experiment, run in the open. Updated 2026-11-17.
Question. How many agents can one Everruns installation run at the same time, and what does each one cost?
Answer so far. About 72,000 chat agents on one 48 vCPU machine on EKS, with a cell node and a quarter of the executors killed in the middle of the run: 720,990 turns, 0 errors, about 200 turns per second, and roughly $0.00003 of compute per agent-hour. Pushed past memory capacity on purpose, cells turned work away and recovered with 0 errors, and a killed node rejoined in about 3 minutes. Models were simulated, so this measures Everruns itself, not model latency.
What's next. A v2 org started with chat only; session files, MCP servers (with OAuth tokens the data plane refreshes) sandboxes, schedules, triggers, long tool calls, subagents, Slack, observers and trace export now run on v2 too, and next is one test org fully on v2. The final step gives v2 every feature (files, MCP, sandboxes, schedules, subagents, Slack, observers), runs one full org on it end to end, and repeats the 75,000-agent run with agents that use those features. That result decides the path: everything on v2, keep v1, or a hybrid.
Where we started. Today's design keeps every session in one PostgreSQL database. On a small cluster it handled about 1,000 agents started together, and the database was the wall: reaching 8,000 on that hardware would have needed about 16 times its CPU (measurement).
The design
Everruns splits into one control plane for management and two data planes for running agents. The control plane (server and database) owns orgs, users, agents, secrets and billing. Data plane v1 is today's server-and-workers execution. Data plane v2 is new: cells. Each org runs on one data plane, and a v2 turn never touches the server or the central database.
Why split. An earlier plan kept one shared path and only replaced the session log. It could not reach the goal: every turn still called the server for agents, secrets, usage and budgets, most of them database reads. With a separate data plane:
- Nothing central on a turn's path. A turn gets its agent and sealed secrets from caches the data plane keeps current, checks spend against its cell's local allowance, and writes only its cell's log.
- Capacity grows by adding cells. Cells are independent; nothing on a turn's path crosses them.
- Little coexistence work. v2 orgs never reach the server's session features, so those need no second path.
Inside a cell
A cell is three nodes, one per zone, running one Raft group. Sessions live in replicated memory, the Raft log is the only hot write, and history goes to S3 in immutable segments. Turns run on a shared pool of executors that pull work, so a slow model call never holds a cell node. Idle sessions keep about 20 KB in memory and page their history back in when needed. Sessions, not orgs, are placed on cells, and move between cells live.
Life of a message
When things fail
| Failure | What happens | Data lost |
|---|---|---|
| One node or one zone | Raft elects a new leader; the replaced node catches up from a snapshot and the log | None |
| A cell loses every node | Rebuilt from its last snapshot plus the log its leader shipped to S3 | At most a few seconds of entries not yet shipped; never a started side effect |
| Bad release corrupts state | Point-in-time restore of the cell, or of one org's sessions | Turns after the chosen point |
| Region loss | Restore in a second region from the replicated bucket | Replication lag (target 15 minutes) |
Trade-offs we accept
- We own consensus. Snapshots, membership and moves are ours, so they are proven with formal models, deterministic simulation and kill tests in CI.
- Config is eventually consistent. An agent edit reaches the data plane in under a second; a running turn keeps the version it started with.
- No SQL across a v2 org's sessions. Lists and search come from session summaries; analytics come from an opt-in export of the org's events as hourly files in object storage.
- Feature lag. v2 orgs start with chat; files, MCP and sandboxes come in later stages.
Progress
Every step has a benchmark gate. A step is done when its gate holds, not when its code merges.
| # | Step | Status |
|---|---|---|
| 0 | Benchmark first, core protocols on paper | ✅ Done |
| 1 | Engine hooks: 5 waited writes per turn (was 22), idle session 1.3 KB | ✅ Done |
| 2 | Trusted host extensions | ✅ Done |
| 3 | Agent definitions as data | ✅ Done |
| 4 | Today's workers run sessions on the Engine | ✅ In production |
| 5 | The cell itself: Raft storage, formal models, deterministic simulation, kill tests | ✅ Done |
| 6 | One thin, complete cell deployment | ✅ Done |
| 7 | Control plane contract | ✅ Done |
| 8 | Edge router, live streaming, full local setup | ✅ Done |
| 9 | Disaster recovery | ✅ Done |
| 10 | Milestone: 75,000 agents on EKS | ✅ 72,000 agents clean with kills, overload and rejoin |
| 11 | Milestone: a full org on v2. Every platform feature on cells, one org working end to end, then 75,000 agents using those features | 🔵 In progress: files, MCP servers, sandboxes, schedules, subagents, Slack, observers and trace export run on v2; one full test org next |
| 12 | Finalization: moving organizations to v2, v2 as the default, upgrade path for self-hosted installs, production rollout, metrics, documentation | ⚪ After the decision |
11 of 13 steps done. Step 11 ends with the decision between full v2, v1 or a hybrid; step 12 carries it out.
Numbers so far
| Measure | Result |
|---|---|
| Waited writes per chat turn | 22 → 5 |
| Turns per second late in a long session | 5.4 → ~1,000, flat |
| Memory per idle session | 6.2 KB → 1.3 KB |
| Cell leader CPU per chat turn | 2.75 ms |
| Edit reaches the data plane | p99 9.4 ms |
| Live reply chunk to the browser, 200 streams at once | p99 69 ms |
| Rebuild of a lost cell with 37,534 sessions | 3.7 s |
| 72,000 agents on EKS, one 48 vCPU machine, nodes killed mid-run | 720,990 turns, 0 errors, ~200 turns/s |
| Same, pushed past memory capacity (bursts of 30,000 new agents) | 716,700 turns, 0 errors, no crashes |
| Killed node back and caught up | ~3 min (was ~10 min with 5 out-of-memory crashes) |
| CPU at 72,000 agents | cells 12.5 cores, executors 18.5, machine 69% average |
| Compute per agent-hour (at $0.05 per vCPU-hour) | ~$0.00003 |
What the EKS runs taught us
Each failed run found a real bug, fixed before the next run:
- A fully loaded machine starved Raft's heartbeats, and cells stopped committing. Raft now has its own listener and threads, and a stalled cell sheds load instead of piling up requests.
- A node killed under load could never catch up, because its snapshots were bigger than a request size limit. That limit is gone, and a node far behind steps aside until it catches up.
- Log shipping to S3 went through a small network gateway that ran out of burst after 40 minutes, so every cell stopped taking work to protect its loss window. S3 traffic now bypasses that gateway, and only cell leaders upload snapshots.
- A restarted node forgot how far its log was committed, so it replayed everything it missed in one batch and ran out of memory. It now keeps that point on disk and replays in small chunks.
Read more
- Design, including the scale estimate and its assumptions
- Detailed step log
- Pull requests
- Questions or ideas: Discord
Everruns is a free, non-commercial open-source project by Mykhailo Chalyi and contributors.