Researchers have a new approach to one of the quieter bottlenecks in running large language models: the memory consumed by key-value caches.
As a language model generates text, it stores intermediate computations in a key-value cache that grows linearly with context length. Most existing methods for trimming that cache rely on static rules or rough proxy scores to decide which entries to drop — and those heuristics tend to break down when token relevance shifts mid-generation. KVpop, introduced in a new preprint, takes a different route: it trains a scorer to directly predict keep-or-drop decisions using a future-attention target, without building the full dense attention maps that would otherwise make this expensive. A second component, a delayed memory-based scorer, waits a fixed number of steps before scoring so it can use near-future context — a wrinkle no other learned eviction method currently offers.
The numbers are hard to dismiss. On Qwen3-4B, KVpop holds 98% of full-attention performance at 75% cache compression and 97% at 88% compression on AIME and HMMT mathematical reasoning benchmarks. Qwen3-8B results are stronger still, approaching teacher performance at comparable compression levels. Memory costs in inference are a real deployment constraint — not a research curiosity — especially as context windows stretch into the hundreds of thousands of tokens.
KVpop is not the first attempt at learned eviction, but if the benchmark numbers hold outside controlled conditions, it represents a meaningful step past the static-heuristic baseline that most production systems still rely on.