When an AI assistant is used across dozens of turns, or over conversations that run for hours, the most common complaint is not "it isn't smart enough" but "the further we go, the more it forgets the rules I set earlier, and the answers keep drifting." The root cause is not the model's intelligence but the fact that the context window is a finite token budget. Even though by 2026 frontier models such as Claude, Gemini, and GPT ship with million-token windows almost by default, a long conversation still burns through that budget: key constraints get pushed out of the window, cost accelerates, and attention gets diluted. Context engineering is the discipline of deciding, within that fixed budget, what to keep, what to drop, what to compress, and where to move it.
The context window is a finite budget: three walls long conversations hit
Every stretch of dialogue history and every document you paste in consumes token quota. As a conversation grows, you hit three walls in turn:
- The hard capacity wall: the window has a ceiling. Exceed it and history must be truncated; the earliest messages are silently discarded, including the rules and limits you set at the start.
- The cost and latency wall: most APIs bill by input token, and every turn resends the entire history. Spend on a long conversation is not linear but accelerating. A big window is not a cheap one either: Google bills Gemini 2.5 Pro requests over 200K tokens at $2.50 per million input tokens instead of $1.25 — filling a million-token window means paying the premium tier on every turn.
- The accuracy wall: research calls it "lost in the middle." Even when the information is still inside the window, models pay the least attention to content placed in the middle. The more noise you pack in, the more easily the key instructions get diluted.
In short, stuffing the window full does not make the AI smarter — it makes it costlier, slower, and more easily distracted. The goal of context engineering is to use the fewest tokens to get the model the right information at the right moment.
"Getting dumber the longer you chat" is not an illusion: the evidence for context rot
Long-conversation degradation is often dismissed as subjective, but there is systematic data behind it. In 2024, teams from Stanford University and UC Berkeley published "Lost in the Middle," showing a U-shaped curve: accuracy is highest when the answer sits at the beginning or end of a long input and collapses in the middle. In 2025, Chroma's "context rot" study widened the scope to 18 frontier models (including Claude Opus 4, GPT-4.1, Gemini 2.5, and Qwen3) and reached the same conclusion: as input tokens grow, every single model degrades, often well before the window limit is reached. In its LongMemEval tests, feeding the same question with roughly 300 tokens of distilled context yielded markedly higher accuracy than feeding the full history of about 113,000 tokens. The study also found that semantically similar but irrelevant "distractor" passages actively mislead the model, repeatedly surfacing in its hallucinated answers.
The crucial implication: this is a structural property of the transformer attention mechanism, not a capability gap that "just switching to a bigger-window model" will fix. Since packing in more content hurts accuracy, managing context is not optional — it is the baseline engineering of any long-conversation system.
Sliding window and truncation: cheapest, but amnesiac
The most intuitive approach is the sliding window: keep only the last N turns and let the rest fall away. It requires almost no extra computation and is the cheapest to implement, well suited to cases like customer-support Q&A where "recent context is usually all you need." The price is total amnesia: once a preference, identity, or constraint the user set at the start slides out of the window, the AI acts as if it never heard it. A classic trap: the user says up front, "quote everything in New Taiwan dollars," but thirty turns later that line is long gone and the AI starts quoting in US dollars again. That is why pure truncation is rarely used alone; it is usually paired with pinning key messages, or summarizing before eviction, so important constraints are not thrown out with the rest.
Summarization and compaction: distilling history into essence
Rather than discarding whole stretches, compress old dialogue into a summary and keep that. A common technique is the running summary: once history reaches a set token threshold, the earlier portions are handed to the model to distill into a few key points, and the summary replaces the original text as the conversation continues. In its September 2025 context-engineering guidance, Anthropic calls this compaction and reveals the principle behind Claude Code's implementation — the summary deliberately preserves architectural decisions, unresolved bugs, and key implementation details while discarding verbose tool outputs and redundant messages.
The cost benefit is clear: the longer a conversation runs, the more "summarize then send" saves compared with resending the full raw history every turn, driving the average per-turn token cost steadily down over time. But summary quality makes or breaks it: repeated compression causes "context drift," where precise numbers and subtle constraints evaporate turn by turn. The key is to explicitly define which fields must always survive — order numbers, customer identity, confirmed decisions — designating them as mandatory-retain fields rather than letting the model freely decide and compress away important premises.
Chunking and offloading: move memory out of the window, fetch it back on demand
When the volume of background material far exceeds window capacity, the right move is not to cram it in but to offload it: split the full history and documents into semantically complete chunks, store them outside the window, and each turn retrieve only the few most relevant chunks back into context. This frees effective knowledge from the window's size limit and directly eases "lost in the middle," because only content highly relevant to the moment enters the window.
Chunking parameters directly shape quality. Practical experience suggests roughly 512 tokens per chunk for general use, with 10–20% overlap between chunks so a key fact spanning a boundary is not cut in two; Microsoft Azure's own documentation recommends starting with 512 tokens and 25% overlap as a baseline. On method, semantic chunking achieved up to about 91.9% recall in Chroma's tests versus 85–89% for fixed-length chunks — yet Vectara's NAACL 2025 study cautions that a simple fixed 200-token split often matches or even beats semantic chunking on real-world data, and after testing across 3 embedding models the study found chunking configuration influences retrieval quality no less than the choice of embedding model. In other words, rather than agonizing over the algorithm, get the chunk size and overlap right. Note that this is about moving the history of "this current long conversation" outside the window for on-demand retrieval — capacity engineering for a single conversation, a different layer from building a long-term memory store across conversations and users.
Pinning key information and combining strategies: don't let compression lose the constraints
Every compression and eviction strategy shares one risk: throwing out something it shouldn't. Context engineering must therefore deliberately preserve a set of "must always be present" facts. A common approach pins system instructions, user identity, and hard constraints to a fixed position in the window, immune to sliding and summarization; a more robust one extracts these key facts into structured state (for example a field table updated as the conversation proceeds), managed separately from the compressible free-form dialogue. The trade-offs of the four strategies:
| Strategy | Best for | Upside | Cost |
|---|
| Sliding window | Short flows needing only recent context | Near-zero cost, simplest to build | Loses the rules and constraints set at the start |
| Summarization | Long conversations that must keep the backstory | Retains large history with few tokens | An extra model call; summary quality decides success |
| Chunk + offload | Background far larger than the window | Effective knowledge not capped by window size | Needs retrieval infrastructure; chunk quality matters |
| Pinning key info | Hard constraints that must never be forgotten | Guarantees core instructions are always present | Occupies a fixed quota; scope must be set manually |
In practice these four are rarely used alone but combined into one pipeline: pin the key constraints, keep recent dialogue verbatim, compress older history into a summary, and offload bulky background to external retrieval. One team combined a sliding window, relevance retrieval, and structured state to cut average tokens per request from about 18,000 to 6,500 — roughly 64% — with no drop in answer quality. The right ratio depends on how long your conversations run, how much cost you can tolerate, and which information can absolutely never be lost.
How Nerdtechnic helps design context engineering
Stable performance in long conversations comes not from switching to a bigger-window model but from context engineering that spends the token budget where it counts. When helping enterprises adopt AI assistants and agents, Nerdtechnic first measures actual conversation length, cost ceilings, and the key information that must be preserved, then designs a combined strategy of sliding, summarization, chunk offloading, and pinning of key information — validating accuracy and per-turn cost with observable metrics. This lets the AI hold its focus and control its cost across a long conversation, without forgetting, at the critical moment, that one rule you set at the very beginning. Manage context well, and AI evolves from "good at chatting" into "reliable even after chatting for a long time."
References
Technical Sharing
