Autonomous Long-Running Coding Agents

The goal is a contract, not a prompt

The first move is to stop talking to the agent turn-by-turn and hand it a goal up front. A good goal states four things: the end state you want, the evidence required to prove success, the constraints it must not violate, and where possible a turn or budget cap.

Saravia's term is that the goal 'works more like a contract than a longer prompt' — this is goal-as-contract, telling the agent the exact end-state plus the proof of success so it can't fake 'done'. The failure mode he names is precise: a weak goal gives the model room to stop early, take shortcuts, or redefine success in a way that looks fine in the transcript but fails in the real system.

The judgment that still matters is human. Encoding the domain knowledge — a target benchmark, a reference screenshot, a baseline to beat — is the part the model would otherwise guess at. The model executes; you still define what 'done' actually means.

Two roles, not one: evaluator and verifier

A long-running agent needs a second role beyond the executor. The evaluator is whatever judges the work — another agent, an LLM-as-judge, a test suite, a benchmark. The design choice is matching it to the task: when success is crisp, use a deterministic check (a pass/fail test like types, unit tests, or lint, with no judgment involved); when success is fuzzy — is this report coherent, does this UI match the design — you need an agent's language and judgment.

The deeper idea is the verifier: an external check the agent cannot talk its way around. 'A coding agent can generate a plan, implement a feature, and explain why it believes the work is complete, but that explanation should not be treated as evidence.'

That is the load-bearing distinction in the whole piece. Proof comes from outside the agent, never from its own narration. The pattern he recommends is layered: cheap deterministic checks as the floor, agent review on top for the judgment-heavy failures.

Where it admits it breaks: the verifier is unsolved

This is the most honest part. Saravia flags an OOD problem — out-of-distribution, meaning a task that falls outside the model's training data, where it 'struggles significantly'. If the verification job you hand the agent is unfamiliar, its checking degrades sharply, so a model grading its own novel work is shaky ground.

'If the verifier is vague, the model will often satisfy the easiest interpretation of the task. If the verifier is too narrow, the model may overfit to it and miss the broader intent.' He is explicit that fine-tuned verifiers are 'still an open area of research'.

That is the warning label to read the rest of the piece against. Code has test suites. A lot of the work people actually point agents at — a written note, a summary, a synthesis — has no deterministic test, so an agent-evaluator is the only option available and it is the weakest link in the chain.

The loop is what keeps autonomy alive

A goal gives direction; a loop keeps the work going. Models 'often stop before the real task is finished' — they hit a turn limit, lose confidence, run out of context, or decide a partial answer is good enough.

The loop is the outer control system: it 'wakes up, inspects progress, runs checks, compares the result against the goal, and sends the agent back in with the next instruction when the goal has not been met.' The simplest version is the 'Ralph loop' — a fixed prompt re-piped until a deterministic condition holds. The richer version puts an evaluator agent in the loop to reason about what should happen next.

The line worth keeping: 'Long-running autonomy works as repeated effort under supervision from a control layer, not as one continuous act of intelligence.' The agent can still fail. The point of the loop is that the system notices the failure and continues, instead of silently declaring victory.

Model choice becomes an architecture decision

Planning stays critical. You can ask a frontier model to draft a plan, but you still inspect it, challenge the assumptions, and sharpen the success criteria before handing it to an autonomous loop.

That produces a useful division of labour: a stronger model defines the goal and structures the evaluation, while a different, possibly cheaper model runs the implementation. 'Engineers should stop thinking of the model as a single choice. Model choice becomes an architecture decision.'

Some models plan better, some execute better, some are cheaper evaluators, some are better at vision-based review. A good orchestrator lets you swap these roles rather than wait for one vendor to ship the perfect agent.

Artifacts you can watch, and sessions you can learn from

Two operating patterns close the piece. The first is visual artifacts as control surfaces: terminal transcripts stop scaling once several agents run in parallel, so 'separate storage from presentation'. Durable evidence goes somewhere the agent can search; a rendered dashboard shows the state a human monitors.

The second is session mining — scanning past agent transcripts to turn recurring failures into rules. 'If an agent repeatedly fails the same way... that pattern should not stay buried in logs.' An agent reads the last month of transcripts, finds the repeat failure modes, and proposes updates to the project instructions. 'The agent forgets, the repo doesn't.'

His closing slogan is the whole thesis in one line: 'using a coding agent gives you a conversation; engineering an autonomous coding system gives you a harness.'

Vocabulary

• evaluator — the second role that judges whether the agent's output meets the goal
• verifier — an external check the agent can't argue past — proof, not its own claim
• deterministic check — a pass/fail test (types, unit tests, lint) with no judgment involved
• OOD (out-of-distribution) — a task outside the model's training data, where its checking degrades sharply
• session mining — scanning past agent transcripts to turn repeat failures into project rules

If you're building — what to watch for

• Write the goal as a contract before the run starts: end state, the evidence that proves it, the constraints, and a turn or budget cap. If you can't state the proof, you don't have a goal yet.
• Never accept an agent's own 'done' as evidence. Give it something external to pass — a test, a benchmark, a reference artifact — and treat its explanation as narration, not proof.
• Layer the checks: cheap deterministic tests as the floor, agent review only for failures that genuinely need judgment. Don't reach for an LLM judge where a lint rule would do.
• Be honest about the work that has no test. Writing, summaries, judgment calls — an agent-evaluator is the only verifier available there, and it's the weakest link. Keep a human on those outputs longer than feels necessary.
• Split the roles across models rather than shopping for one model that does everything: one to plan and define success, a cheaper one to execute, another to evaluate.

Reading it critically

• It's a session recap built around one tool's goal and loop commands. The anatomy is clean, but the evidence is assertion rather than benchmarks — the out-of-distribution claim is offered as 'based on my research', not shown.
• It admits the verifier is unsolved, which makes 'autonomy works when you have a reliable verifier' partly circular. For fuzzy work you can't fully verify, you're back to trust.
• The work most people want to automate — notes, drafts, syntheses — has no test suite, so it lands in exactly the weak, out-of-distribution case the piece flags as unsolved. The advice is strongest where you need it least.
• The operating model (cheap subset first, layered verification, proof artifacts) is real overhead it never quantifies. For a small team or a solo build the harness can cost more than the work it supervises, and there's no test offered for when NOT to build a loop at all.