Portkey

The practice of storing pre-computed Key-Value (KV) states of prompt tokens on model servers to eliminate redundant processing and drastically reduce latency and cost for repetitive context.

A caching system that evaluates user queries based on vector similarity, returning previously generated responses or retrieval results for semantically identical questions.