Researchers have a new approach to one of the more stubborn costs in running large language models: the memory eaten by key-value caches at long context lengths.
DepthWeave-KV, described in a new arXiv paper, compresses key and value states by factorizing them across neighboring transformer layers using shared low-rank channel bases. The twist is that it doesn't treat all tokens equally — a built-in router allocates higher reconstruction fidelity to tokens the model needs most, like instructions and retrieval-critical content. It tracks compression error in real time via attention-output probes, adapting on the fly during generation with no retraining required. A fused CUDA kernel handles basis lookup, dequantization, and attention projection together to cut memory traffic at decode time.
On benchmarks including LongBench, Needle-in-a-Haystack, and L-Eval, the method reaches 8.3x KV memory reduction and 72.8 tokens per second at 64K context while holding near-full-cache quality — outperforming prior compressed-cache approaches on both average score and retrieval accuracy. That matters because memory bandwidth, not compute, is increasingly the bottleneck when deploying long-context models at scale.
Most prior KV cache compression schemes apply a flat budget across all layers and tokens — a blunt instrument that hurts quality when the model needs to retrieve specific facts buried in a long input. DepthWeave-KV's per-token routing is the meaningful differentiator here, though whether it survives contact with production workloads beyond the reported benchmarks is the question any infrastructure team will want answered before shipping it.