AI coding harness: what the evidence says about runners, controls and public benchmarks

Across 176 matched settings in a study submitted on 17 September 2026, researchers found no single best coding harness: results changed with planning, action space and context budget, while shipping products automate different parts of the loop.

Harness design moved from a research loop to product controls

Harnesses did not arrive as one product category. The sequence matters because each step shipped a different slice of the idea.

On 6 October 2022, the ReAct paper framed the loop many coding runners still follow: interleave reasoning and acting, then use observations to choose the next step. The authors reported absolute success-rate gains of 34% on ALFWorld and 10% on WebShop over the baselines they compared.

On 9 February 2026, Stripe said its internal Minions coding agents were responsible for more than 1,000 pull requests merged each week. That is a meaningful production-use figure, but the post does not publish the harness design or a benchmark method.

On 10 March 2026, LangChain’s anatomy of an agent harness laid out the checklist product teams now keep rediscovering: filesystems, code execution, sandboxes, compaction, memory files such as AGENTS.md and search.

On 2 April 2026, Birgitta Böckeler’s harness-engineering article argued that coding-agent harnesses should combine feedforward controls, such as AGENTS.md and skills, with feedback controls, such as tests and judges. That was a design argument, not a release note.

On 17 September 2026, An Empirical Study of Harness Design for Coding Agents broke harness design into planning, action space and context management, then tested those components separately. That remains the clearest answer in this source set to the practical question behind “AI coding harness”: which features pay for themselves, and when.

On 18 September 2026, Claude Code version 2.1.277 added AGENTS.md support when no CLAUDE.md is present. One day later, Claude Code version 2.1.278 on 19 September 2026 changed auto mode for Claude API and Enterprise users, and on Bedrock, Vertex, Foundry and gateways, to default to a server-side classifier that “does not charge for classifier overhead”.

The strongest empirical result is still about context, not branding

The September 2026 paper says the best harness depends on model capability and budget, not on a vendor label. Across 176 matched settings, the authors varied planning, action space and context management while holding the execution loop fixed.

The clearest result is about context. The authors write that context management becomes more valuable as the context-window budget tightens, and that most of the gain comes from preventing context-overflow failures. That matters because overflow failures are a harness problem before they are a model problem.

The same study reports that staging rule-based elision before LLM summarisation delivered the strongest overall efficiency among the context-management strategies it tested. Recoverable elision, by contrast, added machinery that models rarely used and did not improve accuracy.

Planning mattered too, but not uniformly. The paper says planning acts as an accuracy scaffold for weaker models and more as a cost saver for stronger ones, with little accuracy change in the latter case.

The action-space result lines up with current product choices. The authors find predefined tools help weaker bash users, while bash-capable models can work effectively with a bash-only interface at substantially lower cost, especially on command-line-centric tasks.

That is why the current market looks fragmented rather than settled. LangChain’s March 2026 essay describes a harness as everything around the model, including system prompts, tools, bundled infrastructure, orchestration logic and hooks such as compaction. Different products then choose different defaults for those layers.

The cited product docs let you compare controls better than outcomes

The source set does support a useful product comparison, but not the one many buyers want. It tells you more about execution controls, context handling, packaging and trust boundaries than about coding accuracy on shared tasks.

Microsoft Agent Framework publishes the richest control surface in the cited primary sources, but the safe claim is descriptive, not comparative. In the .NET harness samples, Microsoft documents planning via TodoProvider and AgentModeProvider, background delegation, file access, loop control, approvals, durable memory, a confined shell, CodeAct, observability and evals. In the Go implementation, Microsoft says the framework is for teams taking agents from prototype to production, with workflows, middleware, checkpointing, restartability, observability, governance and human-in-the-loop control. The same page also states that the Go SDK is in public preview and that declarative agents, RAG, CodeAct and functional workflows are not yet available.

Claude Code appears in this source set only through its changelog, but those release notes still matter because they show which harness ideas have crossed into product behaviour. Version 2.1.277 on 18 September 2026 added AGENTS.md support when no CLAUDE.md is present, and said the feature was “not yet on Bedrock, Vertex or Foundry”. The same entry also fixed a sandboxing bug where sandbox.excludedCommands could exempt an entire compound Bash command, which confirms that sandbox policy is part of the product surface even if the changelog is not full documentation. Version 2.1.278 on 19 September 2026 then changed auto mode for several deployment routes to a server-side classifier that “does not charge for classifier overhead”, and added an Auto mode server row to /status.

ZCode documents packaging and operator workflow more clearly than agent-policy detail. The repository describes ZCode as an AI coding workbench with a desktop app, browser interface and terminal agent. The same page says a single zcode command launches a TUI by default, starts web mode with --web, or passes other arguments to the Agent CLI. Its docs also describe local and remote workspace flows, including SSH and WSL support, token-based web access and packaging for desktop and command-line distribution. What the repository page does not publish is the internal planning loop, context-compaction method or coding-task success rate.

Jev Ultrafast is narrower, but much more explicit about its runtime loop. The README says every observation produces a new element table, the policy picks from operations such as CLICK, TYPE_TEXT, SELECT, SCROLL and WAIT, and only supported operations and targets are offered. The same page says the default loop uses one request per decision cycle and that model output never becomes selectors, coordinates, shell commands or executable JavaScript. It also names several limits, including shadow roots, frames, canvas, uploads, pop-up tabs, nested scrolling and arbitrary keyboard widgets remaining outside this MVP.

These product pages also show how different teams frame trust boundaries. Microsoft’s security notes say several harness providers extend the agent’s trust boundary to external systems, and that every such capability is opt-in and requires explicit developer configuration. The Go repository adds that applications built with third-party systems are used at the developer’s own risk, and that teams are responsible for approvals, boundaries and responsible-AI mitigations. Jev’s README handles the same problem by constraining the browser action surface and rechecking executed targets against the observed page state.

The table below compares what the cited sources actually disclose

The supplied source set does not include primary-source documentation for Cline, OpenCode or Aider, so the comparison below covers only the harnesses we can verify from the cited pages. That still gives a useful map of what vendors disclose publicly, and what they do not.

Source: linked primary-source repository, changelog and paper pages above. This comparison reflects only what those pages publish.

Why context controls and approvals matter more than a brand name?

If you strip the marketing labels away, the cited material keeps pointing to the same operational questions. How does the runner manage context? What actions can it take? Where does code execute? Who approves risky steps? What happens when the model drifts, stalls or overflows?

LangChain’s March 2026 piece answers that from a design perspective. It argues that harnesses need filesystem abstractions, bash or code execution, safe environments, verification tools, memory-file standards such as AGENTS.md, and compaction when context fills. The September 2026 empirical paper then tests part of that agenda and shows that context controls pay off most when budget is tight.

Microsoft’s sample catalog is useful because it exposes those controls as named providers: planning, background agents, file access and loop evaluators. The same source also flags the cost of that flexibility. Background agents, judge evaluators, external skills and summarisation compaction can all extend the trust boundary to external systems, so Microsoft tells developers to vet them before enabling them in production.

Claude Code’s AGENTS.md support in version 2.1.277 is a smaller but telling sign. It turns a memory-file convention described in essays into a product behaviour users can rely on. The same changelog also states that AGENTS.md fallback was “not yet on Bedrock, Vertex or Foundry”, which is exactly the kind of caveat buyers need and vendors often bury.

Jev shows the other side of harness design: narrow the action space and validate execution hard enough that the model cannot freehand browser control. That makes the loop easier to reason about, but it also limits scope to browser tasks the current DOM reader can handle.

ZCode’s repository makes a different trade-off again. It spends its documentation budget on distribution and workspace topology: desktop, browser, terminal, HTTP and WebSocket services, remote connections, build flows and packaging. That is useful if you are evaluating deployment shape, but it tells you little about planning quality or failure recovery.

Public speed numbers exist, but they are not comparable enough for a buying decision

The public evidence remains asymmetric. Jev publishes task timings, protocol-call counts, model configuration notes and scope limits. Microsoft’s sample pages publish capability detail and security caveats, but not task accuracy or cost data. ZCode’s repository publishes runtime packaging and workspace detail, but not benchmark results. Claude Code’s changelog publishes release changes, including one billing detail, but not task-level performance numbers.

Even the strongest public number in this set comes with a warning label. Jev says its six-run comparison involved three repeats of one task on one browser profile, and “not a general reliability benchmark”. That disclosure is useful. It also means you should not treat the 25% median task-time reduction or the 1,092-to-101 protocol-call cut as a general claim about coding-agent quality.

Stripe’s “more than a thousand pull requests merged each week” figure is the opposite kind of evidence. It signals real production throughput, but it does not tell you which harness features drove that output, what review gates applied, or how the system would behave on a benchmark another team could rerun.

The gap matters because “AI coding harness” is usually a buying question, not an architecture seminar. Teams want to know which runner supports their model endpoints, which one offers approvals, how sandboxing works, how repo context is managed, what the operational cost looks like and whether anyone has published a repeatable benchmark. In this source set, those questions are only partly answered.

What the cited evidence supports for real-world selection today?

If your main concern is programmable controls around the loop, Microsoft’s .NET samples show planning, file access, loop evaluators, approvals and a confined shell, while the Go repository adds workflow patterns, middleware, checkpointing, observability and human-in-the-loop support. That is evidence of breadth in the control plane, not evidence of benchmark superiority.

If your main concern is whether memory-file conventions have reached a production runner, Claude Code version 2.1.277 gives a direct yes. If your main concern is an operational billing signal, version 2.1.278’s note that the server-side auto-mode classifier “does not charge for classifier overhead” is the only explicit cost-related statement in the cited docs.

If your main concern is deployment shape across desktop, browser and terminal, ZCode’s repository gives the clearest factual picture in this source set: one launcher, multiple runtime modes, remote workspace support and packaging instructions for both desktop and command-line distribution. What it does not give is public outcome data.

If your main concern is browser-task latency and hard execution constraints, Jev is the only cited product page here with public timings and detailed loop notes. It is also explicit that this is a browser agent MVP with major surfaces still out of scope.

The September 2026 study is the reason not to collapse those findings into a single winner. It says harness choices should follow the model and the failure mode. Weak bash users benefit from more predefined tools. Strong bash users can succeed with a leaner interface at lower cost. Tight context budgets make compaction and elision much more important.

What we could not verify?

The supplied source set does not include primary-source docs or changelogs for Cline, OpenCode or Aider, so we could not verify their model support, pricing, sandboxing, approvals, context controls or repo workflow features here. A like-for-like buyer’s table for those runners would need their own documentation pages or release notes.

There is no public like-for-like benchmark in these sources that runs Microsoft Agent Framework, Claude Code, ZCode and Jev Ultrafast on the same coding tasks with the same model, budget and stopping criteria. The September 2026 harness study is the closest methodological reference point, but the abstract page supplied here does not publish vendor adapters or product-specific results.

There is also no public source here describing ZCode’s internal planning loop, context-compaction method or measured coding success rates. Microsoft’s Go page lists missing features in preview, but it does not publish coding-task accuracy or cost data for the framework. Claude Code’s changelog shows shipped features and fixes, but not a full approvals model or a benchmark suite. The next useful public step would be a vendor-neutral benchmark that exposes the same tasks, models, budgets and review gates across these runners.

Sources

• https://github.com/microsoft/agent-framework/tree/main/dotnet/samples/02-agents/Harness
• https://github.com/microsoft/agent-framework-go
• https://arxiv.org/abs/2210.03629
• https://stripe.dev/blog/minions-stripes-one-shot-end-to-end-coding-agents
• https://blog.langchain.com/the-anatomy-of-an-agent-harness/
• https://github.com/zai-org/ZCode
• https://github.com/browser-use/jev-ultrafast
• https://code.claude.com/docs/en/changelog
• https://arxiv.org/abs/2609.20804
• https://martinfowler.com/articles/harness-engineering.html