A new academic survey argues that KV caches — the memory structures LLMs use to avoid recomputing attention over long contexts — have quietly become one of the most consequential engineering problems in AI infrastructure.
The paper, posted to arXiv, maps more than thirty KV cache management systems across four design axes: locality, lifetime, ownership, and substrate. From those axes the authors derive five architectural archetypes — local-paged, disaggregated-pipeline, shared-store, memory-pool, and hybrid-tier. The finding that stands out: once you fix the workload and hardware, ownership — meaning which component in a distributed system holds and controls the cache — explains most of the remaining design variation between systems. The survey also audits existing evaluation benchmarks and names seven measurements nobody is currently capturing, tying each gap to an open problem area including fault tolerance, speculative decoding, and mixture-of-experts serving.
This matters because KV cache pressure is the primary reason LLM inference is expensive at scale. Serving a single long-context request can require gigabytes of memory, and systems that cannot share or spill that cache efficiently either overprovision hardware or throttle throughput. A clear taxonomy gives engineers a common vocabulary, and the identified measurement gaps signal where the next round of optimization research is likely to focus.
The survey arrives as inference infrastructure has become a genuine competitive differentiator — every major cloud provider and several well-funded startups are racing to cut the per-token cost of serving large models, and KV cache strategy is one of the few remaining levers that does not require changing the model itself.