A new quantization scheme keeps just a handful of attention tokens at full precision and compresses the rest, aiming to shrink memory use in large language models without the usual accuracy hit.
Researchers describe HyQuant, a hybrid-precision quantization framework for the attention mechanism in LLMs. Rather than compressing all attention states to low-bit formats, which tends to degrade output quality, HyQuant identifies a small set of high-impact regions, including recurring vertical-line tokens and a local sliding window, and keeps those in full precision while quantizing everything else. In the prefill stage it runs a dedicated hybrid-precision attention operator; in the decode stage it applies the same logic to compress the KV cache and fuses dequantization directly into the attention computation to save memory and improve hardware efficiency. The team reports the approach holds accuracy nearly steady across multiple models, tasks, and datasets, and has published code on GitHub.
Most prior fixes for attention quantization lean on smoothing techniques to tame outlier values, trading some accuracy for simplicity. HyQuant's bet is that not every attention weight matters equally, so spending full precision on a tiny, identifiable subset sidesteps that tradeoff. If it holds up outside the paper's own benchmarks, it matters most for long-context inference, where KV cache size is a major memory constraint.
Worth noting: this is a preprint already on its third revision, with no independent replication yet, so "nearly lossless" is the authors' characterization, not a verified industry benchmark.