Production traces turn agent failures into regression tests

FuturPulse analysis · 21 October 2026

On 21 October 2026, production traces can now become repeatable AI-agent regression tests: teams can promote a failed session into a case, rerun it before release, and block a pull request when the failure returns. The key choice is whether to replay recorded fixtures or rerun the live agent against a curated trace-derived dataset.

What does regression testing from traces mean?

Trace-based regression testing starts with a real agent session, not an imagined prompt. A trace is the connected record of a user request, model calls, tool calls, intermediate outputs and final result. It matters because agent failures often emerge across several steps rather than in one bad answer.

Latitude’s failure taxonomy identifies tool misuse, context loss, goal drift, retry loops, cascading multi-agent errors and gradual quality loss. Each needs more than an HTTP error check. A useful trace preserves enough context to show the first wrong action and the later customer-visible consequence.

The regression test is the controlled return trip. Capture the troublesome input and relevant state, specify what should now happen, then run the changed agent against that case. A passing release must not merely produce fluent text. It must meet the contract that failed in production.

What was promised and what has shipped?

The market moved during 2026 from generic observability toward a production-to-CI loop. The timeline below separates published methods from software capabilities described as available. A product roadmap is not treated as a shipped feature.

1. 26 March 2026
   Method stated: Latitude published a four-step diagnostic framework: collect traces, cluster failures, find the root cause, then generate an evaluation from the production failure. This is the core operating model for trace-derived tests.
2. 27 March 2026
   Promise and positioning: Latitude’s platform comparison said its product could generate evaluations from real failures and connect detected issues to coding agents through MCP. It also described the full issue-to-pull-request loop as a direction, rather than claiming every step was fully automated.
3. 21 April 2026
   Shipped approach: Comet introduced Opik Test Suites as unit and regression testing for agents. Its argument was that developers need named failures and pass-or-fail cases, not only a quality score.
4. 17 July 2026
   Shipped platform scope: Opik’s repository listed tracing, datasets, experiments, online evaluation rules and a PyTest integration for CI/CD. The repository was marked updated on that date.
5. 8 August 2026
   Practical test pattern: Kevin Tan’s updated guide set out three layers: deterministic unit tests, scored output evaluations and full workflow integration tests. It also advised growing a golden regression set from production failures.
6. 21 October 2026
   Trace-native CI claim: Tracely’s public repository describes freezing a failed trace into a hermetic case, replaying recorded fixtures in CI and failing the pull request if the case regresses.

Which replay model should your team choose?

There are two valid models, and they catch different classes of breakage. Fixture replay holds the old environment still. Live reruns deliberately exercise the current model, prompts and tools against realistic cases.

Tracely describes hermetic replay: it packages the recorded input plus tool and model outputs as fixtures. That makes the run deterministic and avoids model calls during the test. It is strongest when a code change must preserve orchestration, validation, error handling or a previously correct tool sequence.

Dataset reruns retain the production prompt or trajectory as a test case, then call the candidate system again. LangSmith calls this backtesting: historical production data becomes a dataset and a newer version processes it for comparison. This is stronger for model swaps, prompt edits and retrieval changes, because the new system must still perform under present conditions.

This comparison matters because the same incident should often create two tests. Freeze the original sequence for fast CI. Then add a trace-derived live case that tells you whether the revised agent still fulfils the user’s objective when the model is allowed to reason again.

How do you promote one failed trace into a test?

Start with an incident that has a clear user harm or a clear breached rule. Do not promote every odd-looking answer. A test suite filled with vague cases becomes noisy and loses authority in code review.

1. Find the causal trace. Select the complete session, including preceding turns and tool results. For multi-agent systems, include hand-offs and shared-state changes.
2. Name the failure. Write one sentence such as: “The booking agent ignored the user’s Friday exclusion after a calendar conflict.” A label makes later clustering and ownership possible.
3. Choose the assertion. Use deterministic checks for schemas, tool arguments, budgets and forbidden actions. Use an LLM judge or human annotation for nuanced correctness, relevance or tone.
4. Set the baseline. Confirm the old version fails the new case. Otherwise, the case may be unrelated to the fix or too weak to protect it.
5. Require the candidate to pass. Run the case in pull-request CI, and preserve its result beside the commit, prompt and evaluator versions.

Braintrust’s dataset model permits a record’s input to reference a logged trace rather than duplicate it, while retaining optional expected output and metadata. That design is useful when the trace is the evidence and a dataset row is the stable release-test interface.

Keep the assertion narrow. “The answer must be good” is not a release condition. “The final booking excludes Friday, includes every required attendee, and records a successful calendar tool response” gives a reviewer something inspectable.

How should evaluators judge an agent trajectory?

Judge the smallest level that can catch the harm. A span-level check can reject malformed tool arguments. A trace-level check can assess whether a one-turn task completed. A session-level check is needed when the agent must remember a constraint across several turns.

Langfuse stores scores against traces, observations, sessions or dataset runs. That distinction is operationally useful. Put cheap structural checks near the tool call, then reserve more expensive judgement for the full conversation where history changes the meaning.

Use three evaluator types together:

• Code checks: valid JSON, required fields, allowed tools, timeouts, retry limits and policy boundaries.
• Reference checks: compare to a known answer or a required action when the outcome is stable enough to define.
• LLM-as-a-judge: ask a model to apply a written rubric where correctness is semantic, such as whether the agent followed a user’s constraints.

Do not score only the final answer when the risk is in the route taken. An agent can reach plausible prose after querying the wrong account, skipping an approval step or treating an empty tool response as success. The trace lets you test those hidden actions directly.

Which tools offer the clearest production loop?

The tools differ less on trace capture than on what happens after a failure. The decision is whether your team wants a flexible observability base, a managed experiment system or an opinionated route from incident to CI gate.

Tracely is the most explicit fixture-replay option in this comparison. Its public design describes online evaluators, failure clustering, a fail-to-pass contract for promoted cases, recorded fixtures and a command that can return a non-zero CI result. That is attractive where engineers need reproducible incident tests without paying for live inference during each pull request.

Opik is the broader self-hostable platform choice. Its Apache-2.0 repository says the full platform covers multi-step traces, datasets, experiments, online rules and PyTest-based CI evaluation. Pick it when trace-derived regressions belong inside a wider evaluation and monitoring programme.

LangSmith fits teams that want to turn historical logs into datasets, compare a candidate against a baseline and mix offline regression work with online evaluation. Its documentation distinguishes regression testing from backtesting, then describes failed production runs as candidates for future offline cases.

Langfuse is a strong building block for teams that want control over the data model and test logic. Its documentation recommends iteratively improving datasets from production or staging traces, but the team must decide how incidents become owned, named and blocked in CI.

Braintrust suits teams that treat the dataset as a governed asset. Its trace references, versioned data and per-row experiment history make it practical to preserve a production case without copying all of its fields into a separate file.

What should run on every pull request?

Run the tests that produce stable, fast signals on every pull request. That means fixture replays, tool-schema tests, routing checks and a small regression suite of high-severity production cases. These tests should identify the exact case that failed, not only lower an average score.

Run broader live evaluations on a schedule or before high-risk releases. The live suite can use the same trace-derived cases, but should test model changes, retrieval changes and long multi-turn tasks. It will cost more and vary more, so it needs review thresholds rather than blind automation.

The three-layer testing approach separates deterministic unit tests, output-quality evaluations and end-to-end integration tests. That is the right division of labour. A trace regression suite supplements those layers; it does not replace ordinary tests for routing, retries, authentication or tool contracts.

How can teams prevent trace tests from becoming stale?

Every promoted case needs provenance. Store the incident label, severity, date, original trace identifier, agent and prompt versions, tool versions, evaluator version, owner and retirement reason. A regression test without that context becomes hard to trust six months later.

Review the suite when product behaviour changes intentionally. A test may be obsolete because the policy changed, not because the implementation regressed. Mark it retired with a decision record; do not silently delete an inconvenient failure.

Also watch for test gaming. If a prompt learns to satisfy an evaluator while harming real users in a new way, the score improved but the product did not. Rotate newly reviewed production failures into the suite, and retain a human-review sample for cases with ambiguous correctness.

Where does human understanding fit in?

Trace tests verify the system’s behaviour. They do not guarantee that the engineer understands a risky code change. That is a separate release-control problem, especially when a coding agent wrote the patch.

Sphinx takes a different approach: its Git hook asks the developer questions about what a code diff does when an error occurs, and pauses the push until the answer is satisfactory. It is not a trace-regression tool, but it complements one when the failure fix involves retries, idempotency or irreversible external actions.

The useful combination is simple: let trace-derived tests prove that the known production failure stays fixed, then require a human to explain why the change cannot create the same class of harm elsewhere.

What we could not verify?

Public product pages do not establish how often each vendor’s automated judges agree with expert human review in a customer’s domain. They also do not settle retention limits, security terms or operational cost for a specific deployment. Vendors, a customer’s security team and the team operating the agent can settle those questions with contract terms, configuration records and calibration data.