Context Caching

Context caching, or prompt caching, is an infrastructure-level optimization for Large Language Models. When an LLM processes an input prompt, it computes a mathematical representation called the KV cache. Normally, this cache is discarded after the generation is complete. Context caching allows developers to explicitly flag large, static portions of a prompt—such as extensive system instructions, large codebases, or complex tool definitions—so the model provider retains their KV cache in memory for a period of time. From an architectural standpoint, subsequent requests that share the exact same cached prefix can bypass the compute-heavy prefill stage for those tokens. The system only needs to process the newly appended tokens (like the user's specific query). This is typically implemented via API headers or specific structural blocks in the API payload, and cache hits are routed to the specific server nodes holding the state. This is a transformative capability for AI agents and heavy RAG workloads. It enables developers to load massive amounts of background context (e.g., a whole book or thousands of lines of code) once, and then interact with it repeatedly at near-instant speeds. It significantly lowers Time To First Token (TTFT) latency and can reduce input token API costs by up to 80%, changing the economic viability of complex context-engineering patterns.