Researchers have found a way to run the attention mechanism inside large language models at 4-bit precision on Huawei's Ascend NPUs — no retraining required.
The technique, called HiFA4, targets FlashAttention's two core matrix multiplications and executes them as 4-bit HIF4 Cube operations while keeping the softmax state in 16-bit floating point. It bundles two fixes. Smooth-QK redistributes quantization error between the query and key matrices after rotary positional encoding, avoiding per-tile computation at inference time. P-Reordering fuses the softmax normalizer directly into the second matrix multiply, which the authors show eliminates a systematic probability-mass loss they measured across 3.6 million attention tiles in a Qwen3-8B trace. On Qwen3-8B, HiFA4 recovered 37.5% of the accuracy gap introduced by naive 4-bit quantization and cut the rate of predictions that disagreed with full-precision output from 16.3% to 8.2%. On Gemma2-9B, the method stayed within 0.7 percentage points of BF16 baseline. A scheduling analysis projects a 35.4% latency reduction over BF16 by fusing the normalizer into the matrix multiply, though the authors note on-hardware validation remains future work.
Most quantization research targets Nvidia silicon, so a rigorous method benchmarked specifically against Ascend hardware fills a gap that matters as non-Nvidia AI infrastructure becomes harder to ignore. The accuracy-recovery numbers are meaningful: halving disagreement with full-precision output is the difference between a quantized model that is usable and one that is not.
The projected latency gains are the headline number here, but they come with an asterisk — "preliminary scheduling analysis" and "future work" are phrases that should temper enthusiasm until the actual hardware numbers arrive.