Picking the wrong KV-cache strategy can quietly tank your AI serving performance — and a new benchmark makes clear there is no universal right answer.
Researchers tested four KV-cache optimization techniques — KIVI, TurboQuant, SnapKV, and CaM — across multiple task types including multi-document question answering, summarization, and few-shot learning. They ran everything on Llama-3.1-8B-Instruct and Mistral-7B-Instruct-v0.3, measuring output throughput, time-to-first-token, task quality, and how much compression was actually achieved in practice. The core problem the paper addresses: prior work evaluated these methods under incompatible conditions, making apples-to-apples comparison nearly impossible.
The headline finding is that compression ratio alone is a poor proxy for real-world performance. KIVI4 held up best on quality across both models; SnapKV led on throughput for long contexts; CaM showed big gains on certain question-answering tasks but proved erratic elsewhere. That variability matters because production systems rarely run a single workload type — they field a mix, and a technique optimized for summarization may quietly degrade retrieval-heavy queries.
This kind of infrastructure benchmarking is unglamorous but overdue. As context windows stretch to hundreds of thousands of tokens, KV-cache memory has become one of the main bottlenecks in LLM serving — the part nobody talks about in a launch post but everybody fights with at scale. The practical upshot: operators need to profile their own workload distribution before committing to a compression scheme, rather than trusting headline compression numbers from a paper evaluated on different tasks entirely.