Context Compression
Authoritative Definition
Techniques for condensing long text prompts, retrieved RAG passages, or conversation histories into fewer tokens while preserving the core semantic meaning and factual information.
Overview & Technical Description
Context compression acts as an optimization layer between information retrieval and prompt construction. Even with large context windows, passing raw retrieved documents or extensive chat logs to an LLM introduces noise, increases latency, and degrades reasoning quality (the "lost in the middle" phenomenon). Compression techniques mitigate this by stripping out irrelevant information before the context is fed into the final generation model. Architecturally, context compression can take several forms. Extractive compression uses small, fast language models or cross-encoders to identify and extract only the most relevant sentences or paragraphs from a larger retrieved chunk based on the user's query. Abstractive compression involves prompting a smaller model to rewrite and summarize the documents. Advanced methods include token pruning (removing stop words or unnecessary characters) and utilizing specialized models like LLMLingua to compress prompts by identifying and preserving only high-information tokens. By employing context compression, AI agents and RAG pipelines can process a much wider net of initial information, compressing the best parts into a tight, highly relevant context window. This directly improves the LLM's attention mechanism on the most critical facts, reduces overall API costs, and speeds up response generation.
Editorial Notes
Aggressive compression can lead to information loss or hallucination if critical nuances or negative constraints are stripped away. Production systems must carefully tune compression ratios and often use specialized, fine-tuned models for extraction rather than relying solely on abstractive summarization, which adds latency.