How pi Designs an Agent That Can Run for a Long Time

AI

How pi Designs an Agent That Can Run for a Long Time

earendil-works’ pi is an open-source agent harness and coding agent, sitting at 95K+ stars on GitHub. It’s fair to call it something like the progenitor of the agent harness space — tools including openclaw were written under its influence.

earendil-works is now rewriting pi’s runtime. The new design lives in a document called harness-v2.md — 3,446 lines, laying out exactly how this v2 rewrite works.

The first few hundred lines set constraints. One example: old v3 JSONL session files must still open and restore to idle; that’s the only backward-compatibility requirement. Everything else, including format, API, and storage layer, is free to be rebuilt from scratch. No migration scripts, no schema versioning.

The remaining 3,000-odd lines are almost entirely answering one question: if an agent’s process dies halfway through a run, how does the new process that picks it back up know exactly where it left off? What’s already done and must not be redone, and what’s unfinished and still needs to happen?

“Just save the messages” isn’t enough

If you’ve built agent applications, you’ve probably run into some version of these:

• The model dies mid-response. How do you resume it after restart?
• A tool call actually succeeded, but the result never made it to disk before the crash. Rerun it, and does the side-effecting action happen twice?
• Most providers do prefix caching on requests: as long as this request’s message sequence matches the previous one from the start, that shared prefix doesn’t need to be recomputed, so it’s fast and cheap. But if recovery inserts a message into the middle of the conversation, everything after that insertion point, even content that hasn’t changed at all, gets billed at full price on the next request. Who eats that cost?

Just saving messages can’t answer any of these. pi’s definition of a durable run is a lot stricter:

An accepted prompt is a durable operation. After a crash, a new process restores the session. It resumes the run from the last safe boundary. Every state that a crash can produce is recoverable.

An accepted prompt is a durable operation. Whatever state a crash can produce must be recoverable, and the new process must be able to pick the run back up from the “last safe boundary.” At any point in time, the system has to know exactly what’s already on disk and what isn’t.

Write the intent, then act

Getting one side-effecting action to reconnect after a crash, no matter where it dies, is easy to say. What actually makes it hard?

If the action never started before the crash, easy: treat it as if it never happened, do it again. If the action fully completed and the result is on disk, even easier: skip it. The hard part is the middle. The action completed, but the record of that fact hadn’t been written yet. How does the new process tell the difference? Guessing wrong in either direction is bad. Treat it as not-done and rerun it, and a side-effecting action might fire twice; treat it as done, and the result might not exist at all. (Backend engineers will recognize this one: it’s what happens without transactions.)

Section 5 of the document answers this:

Before an effect: write an intent record that names what will happen and the ids it will produce. After the effect: append the result as an entry with exactly those ids.

There is no multi-record atomicity and none is needed. Each record and each entry is durable alone. A crash between intent and result leaves the intent unfulfilled; recovery decides per intent type: complete it, retry it, or close it with a synthetic result. An intent is fulfilled if and only if an entry with its provisioned id exists.

Before any side-effecting action, write down an intent: what’s about to happen, and which id the result will land under. Once the action finishes, write the result as an entry under that same id.

There’s no transactional guarantee between the two records, and none is needed: each record is durable and self-contained on its own, chained together by id. What happens if a crash lands between intent and result? It’s handled by intent type: complete what needs completing, retry what needs retrying, and if it can’t be recovered, close it out with a synthetic result. Whether an intent has been fulfilled comes down to one check — does an entry with that ID exist?

The key piece underneath this mechanism is the provisioned id: before an entry actually exists, its id is already allocated and written into the intent record. If the id exists but its content doesn’t match what was promised, that’s flagged as corruption.

There’s no atomicity designed across multiple records here. Each record, each entry, stands on its own.

Session’s four layers of state

Say you’re designing persistent state for an agent that needs to recover from crashes. A few things you’d need to nail down first:

• Where does conversation content live, and in what structure?
• Who’s actually doing the work, and how far along are they?
• After a crash, how does the new process know where to pick back up?
• Which pieces of information are “can change anytime, only the latest write counts”?

pi’s answer is four parts.

tree is the conversation itself. Entries are linked by parentId: messages, model/thinking/tool-activation changes, compaction summaries, branch summaries. The tree is shared and passive, and it only grows; once an entry is written, it's never changed or deleted.

lanes are where the actual work happens. A lane is a name plus a leaf entry, and later work extends from that leaf. Every session has a lane called main; the application layer can create more as needed, keyed by something stable like a Slack thread id.

lane operation logs record the execution process: operation started, step attempted, tool started, message queued, operation finished. Nobody reads this during normal execution. It exists for exactly one purpose: after a crash, the new process uses it to pick up a lane’s unfinished work.

global facts are session-scoped key-values where the newest write wins: the session’s name, an entry’s label, that sort of thing.

tree (shared, append-only)          lanes 
a ── b ── c ── d                    main            → d   (op log: …) 
      └── e ── f                    slack:171943…   → f   (op log: …) 
  
global facts: name = "Refactor auth", label(b) = "checkpoint-1"

The closest existing concept to a lane is a git branch checked out in its own worktree: a name bound to a position, advanced by new work, movable to any entry without rewriting history. One place it departs from git intuition: a lane’s move (navigation) doesn’t only go forward. It can jump to any existing entry on the tree.

The tree holds conversation content only, never lane state or orchestration state. An entry’s parent chain, once written, never changes. A lane’s leaf moves in exactly two ways: appending an entry, or an explicit navigation. A lane can have at most one open operation at a time; two open at once is corruption.

Three kinds of operations

There are only three things a lane can do.

• Run — an accepted prompt, run through all automatic continuations (tool calls, steering, follow-up, auto-compaction) until nothing is left pending.
• Compaction — replaces old context with a summary entry.
• Navigation — moves the lane’s leaf to an existing entry, optionally leaving a branch summary behind.

A run is a sequence of turns; a turn is an assistant step plus the complete tool batch that message triggers. A step is the retryable unit underneath that: it produces an assistant message, a compaction summary, or a branch summary. Its retry count is written to disk, not held in memory.

Retries are routine in an agent loop. Persisting the retry count is not routine. The document puts it bluntly:

a crash-restart loop cannot reset it.

That line guards against a classic failure: a request keeps failing, the process crashes and restarts, the in-memory count resets to zero, it fails again, and restarts again. The loop never reaches its retry cap and never stops burning tokens. Once the count is written into a step_attempt record, a crash-restart loop can no longer zero it out. The retry cap holds across however many restarts happen.

Context only grows at the tail

Across the requests of a lane, provider context only grows at the tail. An insertion before the previous request’s tail invalidates the provider’s KV cache from that point on and multiplies token cost.

Across a lane’s consecutive requests, context only grows at the tail. Insert content into the middle of the previous request, and the provider’s KV cache is invalidated from that point on, and token cost multiplies.

So if a step is in flight and something needs to be written into the conversation, it doesn’t get inserted at its logical position right away. It’s recorded as a deferred write instead, pushed to the next checkpoint, and applied at the tail all at once. Section 6 has this trace:

R   step_attempt          request in flight, context ends at user message U 
    session.appendMessage(M)   caller resolves here 
R   write_deferred         full payload, provisioned id 
E   assistant message A    provider cached [.., U, A] 
E   message M               checkpoint applies the write; tail append

Insert M directly the moment it’s requested, and you get [.., U, M, A] — a sequence that's valid to send a provider, but it invalidates the KV cache starting at M, and it leaves the transcript with a false impression: that A was generated after seeing M, when it wasn't. The checkpoint sidesteps both problems in one move.

Note: Compaction is the one deliberate exception to this rule: one full cache invalidation, traded for a smaller context. Whether that trade is worth it, the document doesn’t dodge; it puts the cost right on the table.

Overflow gets exactly one chance

There’s a kind of “crash” that isn’t the process dying. It’s the output getting cut off mid-stream. One of the stop reasons a model can return is called length, meaning "generation was halted before it finished." That phrase alone is ambiguous: it could mean two very different things underneath:

• Output hit exactly the cap you set (say, maxTokens: 4000), and it stopped there because the model really was cut off by that limit — this is a genuine end. Compacting context can't save it.
• Output got cut off before reaching that cap — this is context-window pressure, or a truncation on the provider’s side, and this case is recoverable: compact the context a bit and retry.

The way to tell them apart is by comparing how many tokens were actually generated against the cap you originally set — how close they land — and not the number actually sent to the provider in the request. Some providers reject an explicit output cap outright; for others, pi just clamps it to whatever context remains. Either way, “the value that’s actually sent” stops being the reference point you want. Here’s the logic:

function isRecoverableLength(message: AssistantMessage, desiredMaxOutput: number): boolean { 
  if (message.stopReason !== "length") return false; 
  if (desiredMaxOutput > 0 && message.usage.output >= desiredMaxOutput) return false; 
  return true; 
}

In plain terms: hit exactly the cap you set, that’s a real end, nothing to do. Get cut off before the cap, that’s a signal it can be saved.

If the response is recoverable, pi discards it outright: no entry ever gets generated. Then it compacts the context and retries. But that chance only gets given once:

One recovery per conversational input. An overflow compaction may start only when no overflow-reason compaction step_attempt is newer than this run's newest consumed conversational message.

Every user input (whether it’s a normal prompt or a steering message) gets exactly one recovery chance like this. Compact, retry, and if overflow happens again, pi doesn’t compact a second time — it writes a give-up error entry directly and fails the run. What this guard prevents is a compact-still-full-compact-again loop spinning forever — only a fresh user input clears the “already recovered once” marker.

Five crash points inside one tool call

Tool calls carry the heaviest side effects, and one call is broken into five numbered crash points:

E   assistant message, calls c1, c2 
X1  before before_tool                nothing durable for c1 
H   before_tool(c1) 
X2  decision made, nothing written    same as X1 
R   tool_started(c1) 
X3  tool executing 
H   after_tool(c1) 
X4  hook interrupted                  same durable state as X3 
E   tool result c1 
X5  result durable                    c1 finished

At X1 and X2, nothing’s on disk yet — rerun the whole path and you’re fine. At X5, the result entry already exists — just skip it. The tricky part is X3/X4: tool_started is on disk, but whether the tool's side effect actually happened isn't information the system has.

pi’s answer is to let each tool declare its own replay: "never" | "safe", and that declaration gets snapshotted into the tool_started record at the moment of execution. Recovery only reruns the call when both the snapshotted declaration and the current code's declaration say safe at the same time:

The tool’s declared replay safety, snapshotted at execution time. Recovery re-executes an unfinished call only when this field AND the current tool declaration both say “safe”; otherwise it writes a synthetic “interrupted” result.

Both sides need to agree because tool implementations change over time — a script marked safe six months ago might not be idempotent anymore.

A suspended lane looks identical to a crashed one

Not every provider response comes back instantly. Batch APIs, background: true requests, and similar cases return a handle right away, and the actual result might not be ready for hours.

The message carrying the handle gets persisted first, the lane suspends, and prompt() returns "suspended" immediately. Later, possibly from an entirely different process, something calls resume(), redeems the handle, and the real result gets appended as a normal follow-on message.

The suspended lane is indistinguishable from a crashed one in storage: an open operation whose newest entry is a deferred assistant message with no successor.

At the storage level, a suspended lane looks exactly like a crashed one: both are an open operation whose newest entry is a deferred assistant message with no successor. Recovery logic never has to specifically ask “did this crash or is it just waiting.” Restore lists both as suspended, and resume() simply checks whether the handle has been redeemed yet.

The ledger gets written first, correctness sorted out after

Every provider request, success or failure, writes a usage record the moment it settles — before any classification or retry decision.

cost durability must not depend on result durability.

The whole point of a retryable step is that the response it produces sometimes never becomes an entry at all — failed attempts, exhausted retry series, discarded overflow responses. If cost tracking depended on that entry existing, the spend would vanish along with the discarded response. So usage gets its own record, written unconditionally.

There’s one gap that can’t be patched, and the document admits it outright: if the transport dies between a response settling and the usage record being written, that money is gone. They call this the irreducible window.

Races only ever resolve two ways

Long-running operations naturally happen concurrently with new input. Say a user drops “focus on the tests” into Slack right as the current turn is about to finish — which one wins?

The fix is structural: every lane has its own FIFO promise queue, called the lane mutation line. Any operation that has to look at state before deciding gets done inside one job on that queue — validate, at most one durable write, update in-memory state. Provider requests, tool execution, hooks, backoff — none of that is allowed inside the job.

Because jobs run one at a time, two concurrent operations on a lane have exactly two possible histories — [A, B] or [B, A] — and both are defined outcomes. No third, interleaved history exists.

Because jobs execute one at a time, any two concurrent operations on the same lane have exactly two possible orderings — [A, B] or [B, A] — and no third, interleaved history exists.

There’s a 12-row race catalog to go with this: prompt() vs prompt(), steer vs run finish, abort vs queue consumption, each row naming its two legal histories and the mechanism that produces them. One row is honest about a case ordering alone can't solve: abort colliding with an in-flight provider or tool side effect. The external effect might have already really happened, and the result just never made it back. The answer here is the same as for a crash: the intent record, plus the replay policy.

Recovery checks itself

Restore is read-only: it never appends anything, never starts any effect. It begins with indexed lookups rather than a full scan: findOpenOperations returns unfinished operations newest-first, scoped strictly to that one lane's unfinished work.

The self-check mechanism is the part I found most striking:

writer/reducer drift is caught the moment it happens instead of one crash later.

Every time resume() finishes, the harness re-derives the storage state from scratch and compares it against the state it's been maintaining live in memory. Any mismatch is corruption, and it faults right there.

Recovery is idempotent by construction too: hit a provisioned id that already exists, and it’s skipped. Crash again mid-recovery, and there’s just less left to recover.

Testing by brute-force enumeration of every crash point

Section 19 of the document splits the testing strategy into three tiers, and the third one is what turns this design from “written in a document” into “provably true.” In drive: "manual" mode, the harness pauses before every single effect and hands the test code a description of what's about to happen; the test decides when to let it proceed.

Crash simulation is just calling close() at a chosen boundary, then reopening the same backend and calling resume(). The crash points aren't hand-picked:

drive each section 6 trace in manual mode, snapshot the backend after every executeAction(), then reopen every snapshot and resume() — and run recovery twice per snapshot, proving half-completed recovery is safe. New effects added to a trace get crash coverage automatically.

Run every trace from section 6 through manual mode, snapshot the backend after every single executeAction(), reopen each snapshot and resume() it, and run recovery twice per snapshot to prove recovering from a half-finished recovery is also safe. Add a new effect to a trace, and it automatically inherits full crash coverage.

Production and tests run the exact same code — the only difference is drive mode. Manual mode exists purely so tests can drive the whole thing one step at a time.

The cost

A single ordinary conversational turn with a tool call writes: a user message, a step_attempt, an assistant message, tool_started, a tool result, and another step_attempt. Every one of those is a durable write, before the model has even finished replying.

The storage layer inherits real complexity too. SQLite’s branch cache has four distinct cases, and the worst one requires copying an entire path. JSONL has to handle a torn tail: the last line cut off mid-write, an append that was never confirmed. There’s a hard single-writer constraint: only one process can write to a session at a time; concurrency is handled through multiple lanes, not multiple processes. fsync isn't even guaranteed by default: a hard power loss doesn't promise the data survives. If that guarantee is ever needed, it gets added later as an explicit capability.

The Non-goals section is just as blunt: aside from v3 compatibility, no migration, no support for multiple writers, no replication.

Whether it’s worth the tradeoff depends on the workload. For an interactive back-and-forth, probably not: the extra write on every step is pure overhead. For a batch job that runs for hours, the cost of a crash-and-rerun far outweighs one extra disk write per step, and the math works out. This design is betting on the latter.

Closing

This spec’s own rollout is broken into dozens of work packages — F0, R0–R3, J0–J6, H0–H8 — each one small enough for a single contributor to claim. The repo already has entries like "Reserved: I2 by @vegarsti."

Here’s the part I find genuinely interesting: a document specifically about “how do you keep a long-running agent from losing work” ships itself the exact same way — cut into small pieces, handed out to many people, each one claiming a slice and building it out. It doesn’t matter who finishes their piece first, and it doesn’t matter who’s not done yet — if it drops, you just pick it back up from the record.

Source: earendil-works/pi · harness-v2 · packages/agent/docs/harness-v2.md