We spend a lot of time asking which AI model is best at writing code. That is only half the question.
The model does the reasoning, but the harness decides what the model can see, how it plans, which tools it can use, when it should ask for help, how it checks its own work, and what happens when something goes wrong. In practice, those decisions often determine whether an agent completes a long task cleanly or wanders into a confident dead end.
Lilian Weng’s recent article, “Harness Engineering for Self-Improvement” , offers one of the clearest frameworks I have seen for thinking about this layer. I used her recommendations to compare five prominent coding-agent harnesses: Codex, Claude Code, Pi, Amp, and Hermes.
The short version: Claude Code and Codex form a leading tier, but for different reasons. Hermes is the most interesting learning-oriented design. Amp is a strong distributed operator. Pi is the cleanest minimal foundation, although it deliberately leaves more of the control plane to you.
One important caveat before we get into the numbers: these are architecture-alignment scores, not a claim that one product will always write better code. There is no public study that holds the model, repository, task set, token budget, permissions, and tools constant while changing only these five harnesses.
What a harness actually does
Weng describes a harness as the system around the model. A good harness turns an open-ended request into an inspectable loop:
Goal → plan → act → observe or test → diagnose → improve → repeat
That loop sounds simple. Making it reliable is not.
The harness has to manage a few different kinds of state at once:
- The current plan and the condition for stopping
- The files, tools, terminals, browsers, and services the agent can touch
- Short-term context and long-term memory
- Subagents and background jobs
- Tests, evaluators, review evidence, and failure traces
- Permissions, sandboxes, credentials, and human approvals
Weng’s larger point is that improvement should move beyond prompt tweaking. Her progression is useful shorthand:
Prompt → structured context → workflow → harness code → optimizer code
Most coding agents are now good at the first three stages. Very few have safely closed the loop on the last two.
How I scored the five harnesses
Weng does not publish a product scorecard, so I translated her design patterns into nine dimensions. The weights emphasize the things that most directly affect reliability in real coding work.
| Dimension | Weight | What I looked for |
|---|---|---|
| Verification and evidence | 17% | Tests, evaluators, blocking checks, and outcome-grounded stopping |
| Context and persistent memory | 15% | Compaction, retrieval, durable state, provenance, and memory lifecycle |
| Safety and human oversight | 15% | Sandboxing, approvals, least privilege, and clear control boundaries |
| Workflow orchestration | 14% | Planning, iteration, recovery, branching, and explicit stop conditions |
| Parallel delegation | 10% | Inspectable subagents, background jobs, steering, and isolation |
| Tools and runtime interfaces | 8% | Tool coverage, protocols, typed state, and integration quality |
| Adaptability and self-improvement | 8% | Learning from outcomes and safely improving the harness itself |
| Long-horizon stewardship | 7% | Maintainability, ownership, observability, and repository health |
| Simplicity and efficiency | 6% | A bounded, general design that avoids unnecessary complexity |
Each dimension received a 0–5 score. Native, default behavior counted most. First-party opt-in features received partial credit. Community extensions demonstrate what is possible, but I did not use them to inflate the score of a stock installation.
The weighted totals are:
| Harness | Alignment score | Best fit |
|---|---|---|
| Claude Code | 90.5 | Workflow-as-code, verifier hooks, and large inspected fan-out |
| Codex | 89.3 | Default containment, open-core runtime, and consistent policy enforcement |
| Hermes | 83.6 | Persistent learned memory, procedural skills, and durable agent state |
| Amp | 76.2 | Remote threads, distributed execution, and supervision across devices |
| Pi | 66.2 | A small, auditable, provider-neutral kernel you can shape yourself |
The 1.2-point difference between Claude Code and Codex is smaller than the uncertainty in the method. Treat them as a joint first tier, not a definitive first and second place.
Codex: the strongest default control plane
Codex is built around an open Rust agent core shared across the CLI, IDE, desktop app, cloud tasks, and the bidirectional App Server. Its agent-loop explanation and App Server architecture show a runtime that persists threads and emits typed items for tools, approvals, diffs, plans, and errors.
That shared core is more important than it may sound. A client can resume, fork, or supervise work without inventing a second orchestration layer. Goals provide a stopping condition. Subagents appear as inspectable child threads. The desktop app can coordinate parallel tasks and worktrees. Skills and MCP tools use the same broader policy model.
Codex’s clearest advantage is safety. Workspace-scoped writes, network-off defaults, OS sandboxing, granular approvals, managed restrictions, and protected control directories are part of the normal execution model. The open core also makes the orchestration easier to inspect than a proprietary cloud runtime.
Its memory implementation is thoughtful: lessons are extracted from individual runs before global consolidation. But memory remains opt-in and explicitly non-authoritative. That is the right safety posture, although it stops short of verified learning.
The main gap is self-improvement. Codex can test its output and iterate on a task, but it does not yet mine failures, generate competing versions of its own harness, evaluate them against held-out tasks, and promote only no-regression candidates.
Score signature: 4.9 tools, 4.9 safety, 4.7 workflow, 4.5 context, and 2.8 strict adaptability.
Claude Code: the strongest encoded workflow
Claude Code takes a different route. Its proprietary runtime exposes a broad file-configured extension layer: hierarchical instructions, auto memory, skills, hooks, MCP, subagents, teams, goals, and JavaScript workflows.
The standout feature is dynamic workflows . Loops, branches, intermediate results, and stopping logic live in readable JavaScript instead of being improvised inside the model’s context. A workflow can fan out to 16 concurrent agents and up to 1,000 agents in a run, expose progress and tool activity, pause work, and encode patterns such as “fix until the validator passes.”
That is very close to Weng’s recommendation to make workflow explicit and inspectable. Goals use a separate evaluator after each turn, while blocking hooks can refuse completion until a validator passes.
Claude Code also has the clearest observability story in this group. Auto memory, path-scoped rules, on-demand skills, separate subagent contexts, worktree isolation, checkpoints, compaction reinjection, and OpenTelemetry provide a lot of evidence about what the agent is doing.
The tradeoff is trust boundary. Bash sandboxing is powerful, but it must be enabled and configured to fail closed. Auto memory is model-authored recall, not verified learning. The goal evaluator sees the transcript rather than independently inspecting the repository. The core runtime is also not open for audit.
Score signature: 5.0 workflow, 5.0 delegation, 4.8 verification, 4.8 context, 4.1 safety, and 3.1 strict adaptability.
Pi: the cleanest minimal kernel
Pi is an MIT-licensed, provider-neutral terminal harness built around a small prompt and four default tools: read, write, edit, and Bash. Nearly everything else comes from TypeScript extensions, Agent Skills, prompt templates, packages, the SDK, RPC, or JSON mode.
Pi’s sessions are full JSONL trees. They retain branching, labels, tool calls, reasoning records, token and cost data, export, resume, and fork. Its compaction system keeps structured summaries and tracks file operations without discarding the raw history.
The extension API can intercept tools, change compaction, add state, implement remote execution, create custom UI, add checkpointing, or expose subagent tools. The agent can even edit an extension, reload it, and continue.
That flexibility is also why Pi scores lower as a stock harness. Its own documentation says there is no built-in plan mode, subagent manager, background Bash, permission popup system, MCP client, or todo system. Those are extension or external-orchestration concerns. Pi also runs with the launching user’s permissions unless you put it inside a container or micro-VM.
The 66.2 score is not a verdict on Pi’s ceiling. It means the default installation implements fewer of Weng’s control-plane recommendations. A carefully assembled Pi stack could score much higher—but at that point the custom stack, not Pi’s kernel alone, is the harness being evaluated.
Score signature: 5.0 simplicity, 4.5 context, 4.0 tools and extensibility, 2.0 delegation, and 2.0 safety.
Amp: a strong distributed operator
Amp’s Neo architecture is a multi-model, cloud-backed agent system. Threads are durable and remote-controllable across the CLI and web. Runners and Orbs let agents continue on fresh remote machines. The Owner’s Manual documents skills, plugins, MCP, specialist subagents, custom agents, review, streaming JSON, and SDK surfaces.
Amp handles long-running context well. Automatic compaction keeps threads moving, while queue, steer, cancel, diffs, and remote control give people high-level supervision. Its redesigned read_thread searches original messages, checks later revisions and reverts, and uses compactions only for orientation. Amp says its longest cited thread had 68 compactions and roughly 21 million raw tokens.
The parallelism story has an important wrinkle. Built-in Task, Oracle, Librarian, review, and custom agents are useful, but ordinary subagents begin without the parent’s accumulated conversation, cannot communicate with one another, cannot be guided mid-task, and return only a final summary. Separate top-level threads and Orbs are more inspectable than those internal workers.
The largest gap is governance. Amp does not ask for tool approval by default; plugins, legacy rules, enterprise policy, or remote isolation must add those gates. The production core is closed and not self-hostable. There is also no documented learned-memory system or verifier-gated harness-evolution loop.
Score signature: 4.5 workflow, context, delegation, and tools; 4.0 verification; 2.5 safety and adaptability.
Hermes: the most interesting learning architecture
Hermes Agent is an MIT-licensed, provider-neutral personal and general-purpose agent. It combines bounded profile memory , SQLite session search, self-authored procedural skills , context compression, goals, cron, subagents, durable Kanban workers, plugins, MCP, gateways, and multiple execution backends.
Hermes makes a useful distinction between memory and skills. Compact facts stay available in the prompt. Longer procedures load on demand and can be created or patched from experience. Goals contain an outcome, verification evidence, constraints, boundaries, and a stop condition. Delegation exposes live child status and cost. Kanban adds durable jobs, retries, heartbeats, worktrees, human comments, verification notes, and residual-risk metadata.
This is the richest persistent-state design in the comparison and the closest to Weng’s learning-oriented vision.
It is not yet a fully self-improving harness. The official self-evolution repository marks skill evolution as implemented, while tool descriptions, system prompts, harness code, and a continuous loop remain planned. More importantly, the current optimizer passes a keyword-overlap fitness heuristic into GEPA and holdout scoring . That is a real optimizer scaffold, but it is not robust semantic verification.
Hermes is also unusually candid about its security boundary. Its security policy says that only OS-level isolation is a true boundary and the default terminal backend runs on the host. Smart approvals, credential filtering, checkpoints, and scanners reduce accidents. Untrusted input still calls for a whole-process Docker or OpenShell wrapper.
Score signature: 4.8 workflow, delegation, and tools; 4.75 context; 4.0 adaptability; 3.8 verification; and 3.5 safety.
The runtime designs are converging, but the safety defaults are not
Chart legend: blue = Claude Code, brown = Codex, green = Hermes, purple = Amp, red = Pi.
Claude Code maximizes encoded orchestration. Codex maximizes consistent policy enforcement. Hermes maximizes persistent agent state. Amp sits close to the leaders on workflow, context, delegation, and tools, but its permissive default lowers the safety score. Pi wins simplicity by refusing to bake these layers into the kernel.
That last point is worth emphasizing. A missing feature can be a design choice rather than a technical failure. Pi is easier to reshape because it does less. Codex is safer by default because it does more. The right choice depends partly on which parts of the control plane you are willing to own.
The self-improvement loop is still open
Chart legend: blue = Claude Code, brown = Codex, green = Hermes, purple = Amp, red = Pi.
Claude Code has the strongest current verification machinery. Hermes has the strongest learning substrate. None of the five combines all of the following:
- Causal failure mining from real trajectories
- Diverse proposals for changing prompts, workflows, tools, or harness code
- An evaluator outside the loop being modified
- Held-in and held-out regression tests
- Preservation of negative results and failed ideas
- Automatic promotion only when a candidate creates no regressions
- Human review at the point where judgment actually matters
This distinction matters because “the agent tried again” is not the same as “the harness improved.” A system can get better at satisfying visible tests while becoming worse at generalizing, maintaining the repository, or resisting reward hacking.
The exact dimension scores
| Dimension | Claude Code | Codex | Hermes | Amp | Pi |
|---|---|---|---|---|---|
| Workflow | 5.0 | 4.7 | 4.8 | 4.5 | 3.5 |
| Context and memory | 4.8 | 4.5 | 4.75 | 4.5 | 4.5 |
| Delegation | 5.0 | 4.6 | 4.8 | 4.5 | 2.0 |
| Tools and runtime | 4.9 | 4.9 | 4.8 | 4.5 | 4.0 |
| Verification | 4.8 | 4.4 | 3.8 | 4.0 | 3.25 |
| Safety | 4.1 | 4.9 | 3.5 | 2.5 | 2.0 |
| Adaptability | 3.1 | 2.8 | 4.0 | 2.5 | 3.25 |
| Stewardship | 4.2 | 4.5 | 3.5 | 3.25 | 3.0 |
| Simplicity | 4.0 | 4.3 | 3.25 | 3.5 | 5.0 |
Readers who disagree with the weights can recompute the total from this table. A low score with strong public evidence may be more trustworthy than a high score supported only by a vendor assertion.
How strong is the public evidence?
I graded the evidence separately from the capability score:
- E0: unknown
- E1: asserted
- E2: documented
- E3: inspectable or reproduced
- E4: demonstrated in a controlled comparison
No harness receives E4. Codex, Pi, and Hermes expose substantial source code. Claude Code documents its mechanisms well, but its core is proprietary. Amp’s strongest public evidence is first-party documentation and product announcements.
High evidence does not mean high capability. Pi’s low stock-subagent score is high-confidence precisely because its own documentation says the feature is not built in.
What I would choose for different kinds of work
There is no single correct harness. The practical question is: which missing control-plane pieces are you prepared to build and maintain?
- Choose Claude Code when workflow-as-code, large inspected fan-out, blocking verifier hooks, and deep observability matter most. Enable fail-closed sandboxing and managed permission rules.
- Choose Codex when default containment, an open reusable core, granular policy, typed execution state, and app-level parallel supervision matter most. Treat memory as recall, not policy, and add independent regression gates for long goals.
- Choose Hermes when persistent learned memory and procedural skills are the priority. Turn on write approvals, isolate the whole process for untrusted inputs, and replace heuristic evolution fitness with task-specific held-out evaluators before automatic promotion.
- Choose Amp when durable remote threads, mobile or web supervision, specialist agents, and disposable remote machines are the differentiator. Add permission plugins or enterprise policy and prefer Orbs for risky work.
- Choose Pi when you want a small, auditable, provider-neutral kernel and are willing to own orchestration. Put it in a hard sandbox, then add delegation, permission gates, verifier loops, and memory as explicit extensions.
For an organizational decision, I would not rely on this score alone. I would run a matched trial on real repositories: fixed tasks, fixed time and token budgets, declared permissions, repeated runs, hidden regression tests, and a delayed review of maintainability.
Where this comparison is uncertain
A few limitations matter more than the decimals in the score:
- Model and harness effects are entangled. These products do not all run the same models.
- A feature is not a reliability result. Having memory or subagents does not prove accurate recall or good delegation.
- Self-improvement can reward-hack visible tests or model judges.
- Long-term repository stewardship is still under-measured.
- Cloud, desktop, enterprise, and CLI surfaces do not always have the same capabilities.
- This market moves quickly. Version-lock any serious decision and rerun the comparison.
I also ran a deterministic 10,000-draw sensitivity check that varied the metric weights by 0.5–1.5× before renormalizing them. Claude Code ranked first in about 90% of draws and Codex in about 10%. Hermes, Amp, and Pi stayed third, fourth, and fifth in every draw. Removing verification from the score moved Codex ahead of Claude Code.
That sensitivity test measures the stability of this rubric. It is not a probability that one product will outperform another on your repository.
The real competition is moving up a layer
The most interesting conclusion is not that Claude Code edged Codex by 1.2 points. It is that coding agents are converging on capable tool use while remaining meaningfully different in orchestration, containment, memory, and verification.
The next major gains will not come from a cleverer system prompt alone. They will come from making the work visible: explicit plans, durable artifacts, inspectable delegation, hard permission boundaries, independent evaluators, and evidence that survives the agent’s own confidence.
The model may write the code, but the harness determines whether we can trust the process that produced it. That is the layer worth designing deliberately—and the one I expect will define the next generation of serious AI engineering tools.