Working Memory

Working memory represents the transient, highly-accessible state that an AI model actively uses to formulate its current response. Conceptually analogous to a computer's RAM or CPU L1 cache, it operates entirely within the strict limitations of the LLM's context window. It contains the system prompt instructions, the immediate recent turns of conversational history, current user inputs, and any raw data payloads returned by recent tool or API calls. Because every token in working memory incurs computational cost and latency through the self-attention mechanism, managing it efficiently is a central challenge in AI infrastructure. If working memory becomes bloated with overly long conversation histories or massive, unfiltered document retrievals, the model may suffer from the "lost in the middle" phenomenon—failing to recall critical instructions buried inside vast amounts of text. Therefore, orchestrators must employ aggressive context budgeting techniques. In advanced architectures, working memory works in tandem with broader episodic and semantic memory systems. When working memory nears its token limit, background processes trigger memory consolidation: synthesizing older interactions into dense semantic facts or summary embeddings, which are pushed to long-term storage and evicted from the immediate context window. This continuous state-management cycle allows agents to run indefinitely while retaining razor-sharp immediate focus.