Turns out there's no single best way to make LLMs remember what they just said.
A new arXiv paper runs a head-to-head comparison of three KV cache management frameworks: vLLM, InfiniGen, and H2O. KV caches store the key and value vectors an LLM computes at each layer, so it doesn't have to redo that math for every new token. That's what keeps inference cost linear instead of quadratic. But as context windows and concurrent request loads grow, those caches balloon, and each framework attacks the memory problem differently: vLLM leans on tensor offloading, InfiniGen uses speculative scheduling, and H2O relies on token eviction heuristics. The researchers tested all three across varying request rates, model sizes, and sparsity levels, measuring latency, throughput, and memory usage.
This matters because "which inference framework should we use" has mostly been answered by vibes and benchmark cherry-picking, not systematic comparison. Anyone running LLMs at scale is making a bet on one of these memory strategies, often without knowing where it breaks down. The paper's contribution is naming the actual conditions - request volume, model size, sparsity - where each approach wins or loses, which is more useful than a single aggregate leaderboard number.
The unglamorous truth here: the future of cheaper LLM inference looks less like bigger breakthroughs and more like this kind of plumbing work, figuring out which memory trick to reach for and when.