Context Window

The context window acts as the fundamental short-term memory boundary for Large Language Models. It defines the absolute ceiling on how much text, system instruction, retrieved data (RAG), and conversational history can be ingested at once. For instance, models have evolved from 2k or 4k token limits to massive windows of 128k, 1M, or even 2M tokens. While a larger context window permits entire codebases or books to be processed simultaneously, it comes with trade-offs. Processing long contexts quadratically or linearly (depending on the attention mechanism) increases compute time, latency, and monetary cost. Furthermore, models may struggle to retrieve specific facts buried deep within massive context windows—a phenomenon known as the needle in a haystack problem. Thus, optimizing what goes into the context window remains crucial even as theoretical limits expand.