Context Truncation
Authoritative Definition
Algorithmic strategies for dropping older or less relevant tokens when a generated prompt exceeds the strict bounds of a models context window.
Overview & Technical Description
Context truncation is a defensive mechanism employed when the aggregated tokens intended for an LLM—comprising instructions, few-shot examples, RAG documents, and conversation history—surpass the models maximum token limit. Without truncation, API calls to the LLM will fail with a maximum length exceeded error. Various strategies exist depending on the use case. Sliding window or tail truncation removes the oldest conversational turns, prioritizing recency. Middle-out token dropping preserves the beginning (often system prompts) and the very end (the latest user query), discarding the middle where LLMs classically suffer from lost in the middle attention degradation. Advanced truncation may be semantic, evaluating chunk importance rather than relying strictly on chronological order.
Editorial Notes
Simplest fallback mechanism for prompt overflow management. While effective for preventing errors, naive truncation can inadvertently remove critical facts needed for reasoning, making semantic compression often a preferred, albeit more expensive, alternative.