LLM agents should summarize a continuing task before history reaches 20 turns, reset after an error or goal change, and split independent work into separate threads: a 15 November 2026 preprint found full transcripts drove matched state-tracking accuracy from about 60% to 10% within 20 turns, while verified summaries preserved a 52–59-point advantage.
At a glance
- Keeping prior assistant replies cut final success by 21 points for GPT-4o-mini in a 48-task math study.
- The same study measured a 29-point decline for Mistral-7B when its prior replies remained in the prompt.
- Apple reported near-full-cache conversation accuracy at 4–6× KV-cache compression with EpiCache.
- The STALE benchmark found its best tested model reached only 55.2% overall accuracy when old memories needed updating.
- A clinical study found prior-chat bias averaged under 2 points in realistic record retrieval, despite larger effects on exam-style questions.
When should an agent summarize its history?
An agent should summarize when one task continues but the transcript is starting to contain old replies, tool output and repeated planning. A study by Ali Şenol is the clearest recent measurement: on 100-turn state-tracking dialogues, full-history accuracy fell from about 0.6 to 0.1 within 20 turns. That is not a billing problem alone. It is an accuracy problem.
Use a summary that records state, not a vague recap. State means the current goal, confirmed facts, unresolved questions, constraints, decisions, source references and the next action. Tokens are the chunks of text a model reads and bills by. A good summary removes old wording while retaining facts that the next decision depends on.
The important qualifier is “verified.” Şenol’s comparison used a solver-verified state summary, not a free-form model recap, and the summary held a 0.52–0.59 accuracy advantage over full history. In other words, a summary can help only if the system checks that it preserved the task’s working state.
| Test condition | Model or dialogue | Measured result | What it means for an agent design |
|---|---|---|---|
| Prior replies retained | GPT-4o-mini | Final success down 21 points | Do not treat the agent’s earlier prose as harmless memory. |
| Prior replies retained | Mistral-7B | Final success down 29 points | History can amplify drift even on structured tasks. |
| Full transcript | CDR-Stream dialogue | Accuracy fell from about 60% to 10% | Use an earlier checkpoint, not an endlessly growing chat. |
Source: The Silent Cost of Conversation History in LLM Reasoning. The rows are different test settings, not a ranking of models.

When should you reset an LLM agent thread?
Reset a thread when its objective changes, when it has acted on a wrong assumption, or when untrusted text has entered the working context. A reset should not mean deleting durable knowledge. It means starting a clean prompt with a short, checked state record and only the evidence relevant to the new task.
The case for resets is strongest when a later event silently invalidates an earlier fact. The STALE benchmark, submitted on 7 May 2026, tested 400 expert-validated conflict scenarios across more than 100 everyday topics. Its best evaluated model reached 55.2% overall accuracy, showing that retrieving a newer fact does not reliably make an agent reject the old premise.
This is the failure mode behind a surprising number of confident errors. The model may remember that a customer preferred one option, then miss that a later message made the preference obsolete. Resetting after a material state change prevents the old statement from retaining equal status with its correction.
Do not reset after every awkward answer. First classify the issue:
- Formatting error: keep the thread and correct the output rule.
- Missing evidence: keep the task, retrieve the missing source and update the state record.
- Wrong premise or changed goal: reset from a verified checkpoint.
- Prompt injection or contaminated tool output: reset immediately and exclude the untrusted text.
When should you split an agent into new threads?
Split a thread when work can proceed independently, needs different evidence, or has different privacy and permission rules. Separate threads reduce irrelevant context and stop one subtask’s mistaken assumptions becoming another subtask’s premises. They also make it easier to audit which source led to which action.
Anthropic calls this broader task “context engineering”: choosing the set of tokens available at each model call. Its September 2025 engineering guide says agent context includes instructions, tools, external data and message history, all of which must be refined as an agent loop grows. That supports splitting work by state and purpose, rather than handing every worker the entire transcript.
A useful split has a narrow contract. One thread can gather sources, another can calculate, and a third can check whether the answer meets requirements. Pass a signed-off output between them: a claim, its evidence, its uncertainty and a timestamp. Do not pass a long private chain of thought or an unfiltered chat log.
Use one thread for a single evolving task with one owner and one source set. Split threads for parallel research, different customers, separate permissions, conflicting goals, or tasks that need independent verification.
Does a bigger context window fix history errors?
No. A bigger context window delays forced truncation, but it does not guarantee that the model will weight every earlier statement correctly. Anthropic warns that recall accuracy can decline as more tokens enter context, a pattern it calls context rot. The relevant constraint is attention: the model must decide which parts of a growing prompt matter now.
Apple’s EpiCache research, published in May 2026, makes the distinction clear. It compresses the key-value cache, the model’s numerical working memory for earlier tokens, into topical episodes. Across three long-conversation benchmarks, Apple reported up to 30% higher accuracy, near-full-cache accuracy at 4–6× compression, and peak-memory savings of up to 3.7×.
That is an infrastructure gain, not proof that every retained statement is true. EpiCache is designed to preserve topic-relevant context under fixed memory limits. It cannot decide whether a customer’s old address, a revoked instruction or an outdated policy should still guide the agent. That decision needs explicit state management.
What does realistic prior-chat bias look like?
Prior-chat bias is real, but its size depends heavily on the task and the distractor. A 2026 factorial study tested 14 models with both MedQA exam questions and 962 record-retrieval questions based on de-identified discharge summaries. Incorrect prior messages reduced multiple-choice accuracy by an average 15.0 points across models in the exam setting.
The same study found a smaller and inconsistent effect in the record-retrieval task: average losses stayed below 2 points across distractor types and response formats. That matters because a synthetic chat that simply states a wrong answer may exaggerate a failure that a real conversation, grounded in a patient record or source file, partly resists.
For builders, the lesson is not that history is safe in realistic work. It is that tests must resemble the work. Measure your agent on its real conversation length, real tool outputs and real error patterns. A single benchmark score cannot tell you when its memory should be summarized or reset.
How should teams test conversation history accuracy?
Test retained state, not conversational fluency. Şenol found that consistency and logical coherence deteriorated much less than actual answer accuracy, while stability rose. An agent can sound steady and internally consistent while tracking the wrong state. That makes self-ratings and polished prose weak early-warning signals.
Build tests around facts that can change. Give the agent a preference, a deadline, a tool result and a later correction. Then ask questions that require it to reject the old state. STALE’s three checks offer a useful pattern: detect that the old fact is outdated, reject a question built on it, and apply the update in later behaviour.
For higher-risk conversations, evaluate more than correctness. Google’s AMIE clinical-dialogue study used 159 case scenarios, 20 primary-care physicians and specialist as well as patient-actor evaluations. Its design is not a general agent benchmark, but it illustrates the right standard: assess the dialogue, the decision and the downstream consequences rather than one final answer.
- Track exact state accuracy at every turn.
- Inject a correction that contradicts an earlier fact.
- Check whether the next action uses the correction.
- Compare full history, a verified summary and a clean reset.
- Record latency and cost separately from answer accuracy.
What should an agent keep outside the transcript?
Keep durable state outside the chat transcript whenever it can be represented as data. That includes user-approved preferences, task status, permissions, source citations, tool results with expiry times, and decisions that require confirmation. Bring back only the fields needed for the next step.
Amazon Web Services’ DynamoDB and Bedrock architecture guide shows the common production pattern: store messages by session identifier, then retrieve history when needed. The missing safeguard is semantic selection. Saving every message makes a chat recoverable, but it does not make every message relevant or valid.
A practical state object can be short:
goal: the current user-approved outcome.facts: each fact with source, time and confidence.open_questions: blockers the agent must resolve.constraints: budget, policy, format and permissions.invalidated: facts superseded by later evidence.
A small open-source project illustrates the limited but useful role of deterministic pruning. The prompt-pruning-layer repository describes removing stale tool state and duplicate context while preserving dependencies. Its public repository had 38 stars and no published release as of its latest listed commit on 29 August 2026, so teams should treat it as inspectable code rather than a validated production standard.
Who benefits from each history-management claim?
Claims about context are not neutral because different organisations sell different parts of the stack. Apple benefits from showing that on-device or constrained-environment cache management can preserve quality while using less memory. Its EpiCache result is useful evidence, but it is a vendor research claim on selected long-conversation benchmarks.
Anthropic benefits when teams adopt context engineering practices that make agent applications more reliable on models such as Claude. Its guidance is valuable operational advice, but it is not an independent comparison of vendors. Amazon Web Services benefits when teams store and retrieve chat histories through DynamoDB and run models through Bedrock; its article is an implementation guide for that stack.
The STALE and Silent Cost papers have a different incentive: their authors are proposing benchmarks and measurements of memory failure. Both are preprints, so their numbers deserve replication before they become deployment rules. The GitHub project benefits from adoption of its pruning design, but it does not claim a peer-reviewed evaluation.
What we could not verify?
No public study here establishes one universal turn count for summarising every model, task and tool chain. The 20-turn result comes from calibrated state-tracking dialogues, not from every customer-support, coding or research agent. Model providers and independent evaluators could settle this with shared, long-horizon tests using realistic tool outputs and public scoring rules.
It is also not public which commercial agent platforms verify their summaries against a source of truth before replacing raw history. Providers could clarify that by publishing their state schemas, reset rules and failure rates. Until then, teams should assume that an elegant summary can omit a critical fact unless a deterministic check or external record proves otherwise.
Sources
- Ali Şenol, The Silent Cost of Conversation History in LLM Reasoning
- STALE: Can LLM Agents Know When Their Memories Are No Longer Valid
- Apple Machine Learning Research, EpiCache
- Anthropic, Effective context engineering for AI agents
- Boards-style benchmarks overestimate prior-chat bias in large language models
- Nature, AMIE conversational diagnostic AI study
- Amazon Web Services, scalable context-aware chatbot guide
- Emmimal prompt-pruning-layer repository

