AI coding benchmarks
On 23 September 2026, SWE-Flux introduced 480 execution-grounded questions across 12 Python repositories; its best tested model reached 37% accuracy, exposing a gap between patch-solving scores and reasoning about what code does when it runs.
A coding agent can produce a passing patch, spend little active time, and still fail the question developers often need answered: what state will this program reach, across tests, inputs and calls
At a glance
- SWE-Flux has 480 execution-grounded instances, and the strongest of five tested language models scored 37%.
- SWE-bench’s original set has 2,294 issue-resolution tasks, while its Verified subset has 500 human-filtered tasks.
- Artificial Analysis records agent wall time, but excludes environment startup, verifier time and other harness overhead.
- Linear cut pull-request waiting from more than six minutes to just over five, despite its test suites nearly quadrupling.
What does SWE-Flux change?
SWE-Flux changes the unit of evaluation from “did an agent make an accepted code change?” to “can a model explain observed execution?” Its answers come from instrumented test runs, meaning test execution records the expected result rather than a person writing an answer or another model judging one. The paper says the set covers control flow, loops, program state, dataflow, exceptions and invariants.
Runtime behavior means what software actually does after it starts. That includes which branch runs, which value changes, whether an exception is thrown, and how one function affects another. Static understanding is reading source code without running it. Both matter, but they answer different questions.
The reported 37% ceiling matters because the models did better on local behaviors than on dataflow, cross-function execution, exact state tracking and combining results across a suite. For a developer, those weaker areas are close to the work behind regressions: explaining a surprising test failure or predicting what a change alters elsewhere.
The narrow finding: a high code-generation or issue-resolution score does not establish that an agent can reliably reason about a running repository. SWE-Flux tests that claim directly; a patch benchmark usually tests it only indirectly.
Which coding benchmarks measure runtime?
The public benchmark market now tests several useful parts of software work, but they should not be collapsed into one claim. SWE-bench asks an agent to generate a patch for a real GitHub issue. Terminal benchmarks test shell-driven work. App-building tests ask for a working artifact. Runtime-reasoning tests ask what execution produces.
The original SWE-bench paper defined 2,294 GitHub issue and pull-request problems from 12 Python repositories. It was designed to test repository-scale change making, including coordination across functions, classes and files. Its headline outcome is issue resolution, not an accuracy score on an explanation of a specific execution trace.
Vals lists coding evaluations for web-app creation, extending an app, code migration, program construction and terminal work. Those categories broaden the view beyond bug fixing. They still do not, on that public catalogue, identify a separate runtime-state metric comparable with SWE-Flux’s execution-grounded question set.
Vellum’s leaderboard presents SWE Bench as “agentic coding” and Terminal-Bench as terminal use. Those labels are useful shorthand. They are not interchangeable with a measurement of whether a model can say which value a variable holds after several calls, or which test sequence produces a failure.
What was promised, and what shipped?
The timeline shows a progression from repository patching toward reproducible evaluation, broader task formats and, this week, direct execution reasoning. “Promised” below means the scope the project publicly stated. “Shipped” means a released benchmark, tool, evaluation harness or reported operational result.
10 October 2023 Scope stated
SWE-bench introduced repository-scale issue resolution: a model receives a codebase and issue, then edits the codebase to resolve it. The stated scale was 2,294 problems. That framed code patches as the central proof of coding-agent usefulness.
27 June 2024 Shipped
The SWE-bench project announced a fully containerized Docker evaluation harness. Containers package a repeatable software environment. This improved reproducibility, but the public outcome remained whether a patch resolves the task.
13 August 2024 Shipped
SWE-bench Verified arrived as a 500-problem subset confirmed solvable by software engineers. That improved task quality. It did not add a separate score for explaining an already-running program.
July 2025 Shipped
The SWE-bench site reported mini-SWE-agent at 65% on SWE-bench Verified using roughly 100 lines of Python. The result highlighted a crucial benchmark fact: the surrounding agent scaffold can be much simpler than the model-facing task environment.
1 September 2026 Shipped
SWE-bench Multimodal v2 became fully open source with 480 tasks for local evaluation. It extends issue descriptions to visual material. That addresses a different missing capability: interpreting the evidence attached to an issue.
21 September 2026 Operational result
Linear reported reducing pull-request wait time from more than six minutes to just over five, while cutting runner time per test by roughly half. This is not a model benchmark. It shows why end-to-end delivery time remains a distinct production metric.
23 September 2026 New benchmark
SWE-Flux was submitted with 480 execution-grounded questions. Its stated contribution is direct dynamic reasoning at repository level, with answers harvested from instrumented tests.
Why is agent wall time not runtime behavior?
Agent wall time measures how long the agent process works on a task. It is a cost and responsiveness measure. It does not measure whether the agent understands the behavior of the application it is changing.
Artificial Analysis defines its time-per-task metric as active agent runtime and explicitly excludes environment startup, verifier time and other harness overhead. That produces a cleaner comparison of model-and-agent work. It also means the number cannot tell a team how long its full pipeline takes after checkout, build, dependencies and tests.
The same leaderboard combines DeepSWE v1.1, Terminal-Bench 4.0 and SWE-Atlas-QnA into a three-benchmark composite. Its own methodology warns that agents with similar index values can differ across repository tasks, terminal workflows and rubric-based evaluations. Runtime reasoning is another axis that should appear beside that composite, not be inferred from it.
That distinction affects buying decisions. A fast agent that misreads state can generate more quick failures. A slower agent that accurately traces an exception path can save a human debugging time. A serious scorecard therefore needs both time-to-attempt and correctness about execution.
What actually waits after an AI writes code?
In a production workflow, code generation is only the opening move. Continuous integration, or CI, is the automated build and test system that decides whether a change can merge. The workflow also includes environment setup, dependency installation, test scheduling, retries, review and release checks.
Linear found that moving workloads to faster third-party runners made jobs 34% faster on average. The same account says its TypeScript compiler workload fell 52%. Those are infrastructure gains, not evidence that an AI agent’s reasoning improved.
Linear also reduced the median change-detection job from 26 seconds to eight seconds. The company achieved that by limiting fetch depth and avoiding checkout where jobs did not need a working tree. An agent leaderboard that omits this wait can rank a fast thinker above a system that delivers slower feedback.
That is the practical gap. Runtime behavior concerns the software under change. End-to-end runtime concerns the delivery system around it. Both can block a developer, and neither is represented by a single “percent resolved” figure.
How do public coding sets compare?
The table combines task counts with the public outcome each set emphasizes. It is an editorial comparison, not a new score. The missing column across the patch-oriented families is a standard measure of whether an agent correctly explains repository execution before it edits code.
| Evaluation set | Public task count | Primary evidence of success | Direct runtime-state questions |
|---|---|---|---|
| SWE-bench Full | 2,294 | Issue-resolving patch | No separate public metric |
| SWE-bench Verified | 500 | Human-filtered issue resolution | No separate public metric |
| SWE-bench Bash Only | 500 | Resolved tasks in one agent environment | No separate public metric |
| SWE-bench Lite | 300 | Lower-cost patch evaluation | No separate public metric |
| SWE-bench Multilingual | 300 | Issue resolution across languages | No separate public metric |
| SWE-bench Multimodal | 480 | Issues with visual elements | No separate public metric |
| SWE-Flux | 480 | Gold answers from instrumented tests | Yes |
Task counts for the SWE-bench variants are from the official SWE-bench leaderboard; SWE-Flux’s count and test-harvested answers are from its paper.
The official SWE-bench family ranges from 300 tasks in Lite and Multilingual to 2,294 in the original benchmark. Different sizes do not make one set inherently better. They describe different trade-offs between breadth, cost, filtering and task format.
What should a runtime-aware leaderboard report?
A runtime-aware leaderboard should keep patch success, execution reasoning and delivery speed separate. Combining them without labels turns unlike outcomes into a confident-looking number. Teams need to know which failure mode a score rules out.
- Patch resolution: did the change satisfy the task verifier
- Execution-trace accuracy: did the agent correctly explain control flow, state and data movement before editing
- Full elapsed time: how long did a developer wait from task start to verified result, including setup and verification
- Active agent time: how long did the model-and-agent process run
- Compute and API cost: what did the attempt consume, with cached and uncached inputs identified
- Failure classification: did the system fail because it misunderstood code, made a bad patch, timed out, or hit the environment
Mini-SWE-agent describes a bash-only agent with linear history and independent subprocess actions. That design makes its trajectory easier to inspect. It is a reminder that benchmark reporting should describe the harness, too: model scores can change when tool access, history handling or execution isolation changes.
The best near-term comparison is not “which model is best at coding?” It is “which system succeeds at this task type, with this agent setup, at this cost, while correctly reasoning about this program’s behavior?” That question is longer, but it is honest.
What we could not verify?
Public SWE-Flux material does not yet establish a shared leaderboard, a standard agent harness, a pricing model, or a published mapping between runtime-reasoning accuracy and real bug-fix success. Its authors, or an independent evaluator releasing reproducible runs, could settle those points.
Public coding leaderboards also do not provide a common end-to-end “developer waiting time” measure that includes environment boot, checkout, dependency setup, test execution, verification and review. Benchmark operators and CI vendors could publish that breakdown. Until then, agent wall time and production delivery time should not be treated as the same metric.
