Shrinking a reasoning model to save compute may cost you more compute than you saved.
Researchers studying post-training quantization — the common practice of reducing a model's numerical precision from 32-bit floats down to INT4 or INT3 — found that compressed models frequently generate longer chains of thought than their full-precision counterparts. The accuracy numbers look fine. The token counts do not. Across math reasoning, code generation, scientific Q&A, and agentic tool-use benchmarks, the study found that quantized models can offset their per-token speed advantage by simply producing more tokens to reach the same answer. The team introduced a metric called the CoT Token Inflation Ratio to quantify the gap between quantized and full-precision reasoning length.
This matters because inference cost is almost never reported as token count — it's reported as latency or price per query, both of which absorb token inflation silently. A deployment team that benchmarks a quantized model on accuracy alone will miss the penalty entirely until it shows up in a cloud bill. The finding also complicates the standard pitch for quantization, which frames compression as nearly free lunch.
Mitigation options tested — prompt engineering and adjusted sampling — produced inconsistent results. Quantization-aware training looked more promising but requires retraining, which defeats much of the convenience argument. The authors' recommendation is modest but pointed: reasoning-token usage should be a required disclosure whenever quantized models are evaluated, not an afterthought.