Shrinking a language model to fit on cheaper hardware may not produce the model you think you're shipping.
Researchers introduced a metric called correctness agreement, which measures whether a base model and its quantized version make the same correct predictions — not just whether their overall accuracy scores match. Testing across multiple models and compression levels from 8-bit down to 2-bit, they found that behavioral divergence shows up even under moderate quantization, at bit-widths where standard accuracy and perplexity numbers still look fine. The team also traced where distortion hits hardest: query and key projections inside attention layers are consistently more sensitive to compression than value and output projections. And the damage isn't gradual — the analysis identified non-linear breakpoints, meaning degradation can arrive suddenly at low bit-widths rather than accumulating smoothly.
This matters because post-training quantization is the default cost-cutting move for deploying large models in memory-constrained environments, and the entire field has been judging it by metrics that apparently miss the point. If a quantized model gets the same questions right on a benchmark but disagrees with the base model on which questions those are, it's a different model — and deploying it as a drop-in replacement is a quiet gamble.
The industry's push toward smaller, cheaper, faster models has outpaced the tooling for evaluating whether those models still behave as intended. Accuracy was always a convenience metric; this research formalizes the case that it's also an incomplete one.