A new quantization method lets large language models run in 4-bit precision without the calibration data that step usually requires.
Researchers describe DualQuant, a post-training quantization framework that compresses both weights and activations to 4-bit precision in one pass, skipping the calibration dataset most activation-quantization methods need. It works by solving the weight-compression problem directly, and the same calculation produces the scaling factors needed for activation compression as a side effect. Tested on Llama and Qwen models against WikiText-2, C4, and zero-shot tasks, DualQuant matched or beat the accuracy of SmoothQuant, a widely-used calibration-based method, and clearly outperformed other calibration-free approaches. On an RTX 4090 running Llama-3.1-8B, a real packed 4-bit implementation delivered up to 2.61x memory reduction and 2.46x throughput speedup over standard 16-bit precision.
Calibration data is a practical bottleneck for teams deploying quantized models: it means sourcing representative datasets and retuning them for every new model, which slows down automated pipelines. Removing that step while still matching calibrated accuracy makes 4-bit inference more practical on newer accelerators built for formats like NVFP4 and MX, which only deliver their speed advantage when both weights and activations are quantized.
The results so far come from one GPU and two model families, so it is worth watching whether the gains hold up across the wider range of open-source models and hardware now shipping with native 4-bit support.