Cells: 75,000 agents in parallel

An Everruns experiment, run in the open. Updated 2026-11-17.

Question. How many agents can one Everruns installation run at the same time, and what does each one cost?

Answer so far. About 72,000 chat agents on one 48 vCPU machine on EKS, with a cell node and a quarter of the executors killed in the middle of the run: 720,990 turns, 0 errors, about 200 turns per second, and roughly $0.00003 of compute per agent-hour. Pushed past memory capacity on purpose, cells turned work away and recovered with 0 errors, and a killed node rejoined in about 3 minutes. Models were simulated, so this measures Everruns itself, not model latency.

What's next. A v2 org started with chat only; session files, MCP servers (with OAuth tokens the data plane refreshes) sandboxes, schedules, triggers, long tool calls, subagents, Slack, observers and trace export now run on v2 too, and next is one test org fully on v2. The final step gives v2 every feature (files, MCP, sandboxes, schedules, subagents, Slack, observers), runs one full org on it end to end, and repeats the 75,000-agent run with agents that use those features. That result decides the path: everything on v2, keep v1, or a hybrid.

Where we started. Today's design keeps every session in one PostgreSQL database. On a small cluster it handled about 1,000 agents started together, and the database was the wall: reaching 8,000 on that hardware would have needed about 16 times its CPU (measurement).

The design

Everruns splits into one control plane for management and two data planes for running agents. The control plane (server and database) owns orgs, users, agents, secrets and billing. Data plane v1 is today's server-and-workers execution. Data plane v2 is new: cells. Each org runs on one data plane, and a v2 turn never touches the server or the central database.

One API, one control plane, two data planes: v1 is today's server and workers on PostgreSQL, v2 is cells with executors and S3

Why split. An earlier plan kept one shared path and only replaced the session log. It could not reach the goal: every turn still called the server for agents, secrets, usage and budgets, most of them database reads. With a separate data plane:

Inside a cell

A cell is three nodes, one per zone, running one Raft group. Sessions live in replicated memory, the Raft log is the only hot write, and history goes to S3 in immutable segments. Turns run on a shared pool of executors that pull work, so a slow model call never holds a cell node. Idle sessions keep about 20 KB in memory and page their history back in when needed. Sessions, not orgs, are placed on cells, and move between cells live.

Inside data plane v2: gateways, cells of three nodes, executors, S3

Life of a message

Life of a message in data plane v2

When things fail

FailureWhat happensData lost
One node or one zoneRaft elects a new leader; the replaced node catches up from a snapshot and the logNone
A cell loses every nodeRebuilt from its last snapshot plus the log its leader shipped to S3At most a few seconds of entries not yet shipped; never a started side effect
Bad release corrupts statePoint-in-time restore of the cell, or of one org's sessionsTurns after the chosen point
Region lossRestore in a second region from the replicated bucketReplication lag (target 15 minutes)

Trade-offs we accept

Progress

Every step has a benchmark gate. A step is done when its gate holds, not when its code merges.

#StepStatus
0Benchmark first, core protocols on paper✅ Done
1Engine hooks: 5 waited writes per turn (was 22), idle session 1.3 KB✅ Done
2Trusted host extensions✅ Done
3Agent definitions as data✅ Done
4Today's workers run sessions on the Engine✅ In production
5The cell itself: Raft storage, formal models, deterministic simulation, kill tests✅ Done
6One thin, complete cell deployment✅ Done
7Control plane contract✅ Done
8Edge router, live streaming, full local setup✅ Done
9Disaster recovery✅ Done
10Milestone: 75,000 agents on EKS✅ 72,000 agents clean with kills, overload and rejoin
11Milestone: a full org on v2. Every platform feature on cells, one org working end to end, then 75,000 agents using those features🔵 In progress: files, MCP servers, sandboxes, schedules, subagents, Slack, observers and trace export run on v2; one full test org next
12Finalization: moving organizations to v2, v2 as the default, upgrade path for self-hosted installs, production rollout, metrics, documentation⚪ After the decision

11 of 13 steps done. Step 11 ends with the decision between full v2, v1 or a hybrid; step 12 carries it out.

Numbers so far

MeasureResult
Waited writes per chat turn22 → 5
Turns per second late in a long session5.4 → ~1,000, flat
Memory per idle session6.2 KB → 1.3 KB
Cell leader CPU per chat turn2.75 ms
Edit reaches the data planep99 9.4 ms
Live reply chunk to the browser, 200 streams at oncep99 69 ms
Rebuild of a lost cell with 37,534 sessions3.7 s
72,000 agents on EKS, one 48 vCPU machine, nodes killed mid-run720,990 turns, 0 errors, ~200 turns/s
Same, pushed past memory capacity (bursts of 30,000 new agents)716,700 turns, 0 errors, no crashes
Killed node back and caught up~3 min (was ~10 min with 5 out-of-memory crashes)
CPU at 72,000 agentscells 12.5 cores, executors 18.5, machine 69% average
Compute per agent-hour (at $0.05 per vCPU-hour)~$0.00003

What the EKS runs taught us

Each failed run found a real bug, fixed before the next run:

Read more

Everruns is a free, non-commercial open-source project by Mykhailo Chalyi and contributors.

index.mdDownload .md