A research team has published ARCQuant, an open-source framework that makes NVFP4 quantization — running large language models at 4-bit numerical precision — accurate enough to rival 8-bit methods without breaking hardware compatibility.
The core problem ARCQuant targets is real: shrinking model weights to 4 bits speeds up inference and cuts memory use, but existing techniques for doing so tend to make messy compromises. Rotation-based methods interfere with how fine-grained formats partition data into blocks; smoothing techniques can't fully absorb the larger errors that 4-bit arithmetic introduces; and mixed-precision workarounds often clash with GPUs that want a single, uniform number format across a computation. ARCQuant sidesteps these by appending quantized "residual channels" to the activation matrix — essentially baking error correction directly into the matrix math so standard, highly optimized GEMM kernels can still run unmodified. The authors show theoretically that worst-case error stays in the same range as MXFP8, an 8-bit format, despite operating at half the bit width.
That's the claim worth watching. If 4-bit inference can genuinely match 8-bit accuracy at scale, the cost of running large models drops substantially — not just for hyperscalers, but for anyone renting GPU time. Tests on LLaMA and Qwen models on RTX 5090 and RTX PRO 6000 hardware showed up to a 3x speedup over FP16 in practice.
NVFP4 is a format Nvidia introduced with its Blackwell architecture, so ARCQuant is currently a Blackwell-only story — useful if you have the newest hardware, less so if you're still on Hopper or older.