A larger context window can hold more tokens, but it does not guarantee that a model can find, distinguish, and use every piece of information with equal reliability. The real problem in long tasks is often context rot: as old assumptions, tool outputs, and irrelevant content accumulate, high-signal information becomes harder to use correctly.
“It fits” and “it works well” are different claims
The original 2023 study Lost in the Middle tested multi-document question answering and key-value retrieval. It found that performance could fall substantially when the relevant information appeared in the middle of a long context, while information at the beginning or end was often handled better. The study also included models explicitly designed to support long context. See the Lost in the Middle paper. This experiment does not prove that every newer model follows the same curve, but it does disprove the simple assumption that anything that fits can be used at equal quality.
Anthropic's 2025 engineering article uses the term context rot to describe how recall can decline as token count grows, and treats context as a finite attention budget. See Effective context engineering for AI agents. Comparing only maximum context-window size when buying a model is therefore like comparing warehouse floor area without considering indexes, labels, or the picking process.
Three common forms of decay in long tasks
1. Contamination from stale information
At turn 10, the provisional definition might be “active user = logged in within 30 days.” By turn 40, it may have changed to “paid within 90 days,” yet both statements remain in history. Without an explicit artifact marking the current version, the model may combine the two definitions.
2. Tool output buries decisions
Repeated show tables results, schemas, query previews, and error stacks rapidly expand context. A critical join key or user-approved condition may occupy only one short sentence and become harder to notice.
3. Conversation becomes the only state store
When the plan, progress, sources, and limitations exist only in chat order, any truncation, fork, or next-day handoff can lose the thread. A longer history postpones this failure; it does not solve it.
Four approaches that work better than endless expansion
Compaction: compress the trace and preserve pointers
Turn old tool results into step summaries while retaining a step_id, cache name, SQL source, or error code so the full detail can be retrieved later. Compaction does not erase detail. It moves detail out of the context that must accompany every turn.
Structured artifacts: move decisions out of chat
Use a Plan for approved definitions, SQL for retrieval logic, and a Report for conclusions and limitations. Artifacts have names, states, and versions, making them more reliable than scrolling up to find a sentence.
On-demand retrieval: fetch detail when needed
Keep only a small summary on every turn. When work reaches a particular table, Reference, or prior step, retrieve the complete content through a tool. This reduces irrelevant tokens, although retrieval can itself miss information. Critical decisions should therefore remain structured and open to human inspection.
Memory governance: retain only knowledge that survives the task
Put “our company uses this fiscal-quarter definition” into approved Memory. Keep “this SQL attempt failed” in the execution record. They have different lifecycles and should not be injected permanently together.
Dividing responsibilities among Plan, Reference, and Memory
| Information | Correct location | Reason |
|---|---|---|
| Denominator, time window, and checkpoints for this analysis | Plan | Bound to the task and requires review |
| Large status-code tables and field mappings | Reference | Retrieve full text on demand instead of injecting it every turn |
| Stable rules used across projects | Approved Memory | Reusable, but must be governed |
| SQL results and errors | Query steps/evidence | Retrievable, but not permanent knowledge |
| Final conclusions and limitations | Report | Becomes a deliverable |
The purpose of this division is to leave only the information required for the correct next step in the context window, rather than every event that has ever occurred.
How Lantide Data handles context in long analyses
The Lantide Data Context Engine supplies a compact Summary on each turn and retrieves full schemas and documents with tools on demand. Long conversations use history truncation, while multi-tool turns can use mid-turn compaction. Recent query steps remain in a concise ledger summary; full SQL is still available from cache sources or step records. See Context engineering and token compaction in the AI Agent architecture.
Analysis content is routed into Plans, References, SQL tabs, Reports, and approved Agent Memory rather than relying exclusively on chat. Long tasks therefore retain external state that people can inspect, and important artifacts remain reviewable.
These mechanisms control risk; they are not a permanent “solution” to context rot. Models, retrieval, and summaries can all omit details. For high-risk analysis, review the Plan before Execute, verify evidence before approving the Report, and start a new conversation or Plan when the context approaches its limit or the task clearly changes direction.
Conclusion
A large context window is valuable, but it is a capacity specification, not a quality guarantee. Long tasks should treat chat as the interaction layer and Plans, SQL, References, Memory, and Reports as the real state. Compaction and on-demand retrieval can then control signal density on every turn. Organize context before buying a larger window.