Trimming model size is the usual play for cheaper AI inference — but a new paper argues the real waste is in the tokens models generate, not the models themselves.
Researchers built a curation pipeline that filters training data for concision and correctness, then applied it to the MAmmoTH-VL single-image dataset. Models trained on the curated data were benchmarked against uncurated baselines and publicly available vision-language models ranging from 1B to 4B activated parameters. The headline result: the curated model achieved a 35x lower "Cost-of-Pass" — measured in FLOPs per correct answer — compared to the most verbose 4B competitor, Qwen3.5-4B, while landing within roughly one percentage point of its accuracy (0.691 vs. 0.704 mean accuracy; 0.41 vs. 14.58 TFLOPs per correct answer). Matched-length accuracy also improved by 17.55 percentage points over the uncurated baseline, a gap that widened as models scaled up.
The finding matters because inference costs compound at scale — every extra token a deployed model generates is compute someone is paying for. The standard toolkit of distillation, pruning, and quantization treats output length as a given; this work treats it as a design variable, which is a meaningfully different frame. The result also challenges the assumption that verbose chain-of-thought reasoning reliably earns its cost: the study found that reasoning-structured verbosity justified its token spend in fewer and fewer capability groups as model size increased.
The paper stops well short of a general prescription — it covers one dataset and a narrow parameter range — but it arrives at a moment when AI labs are under real pressure to justify inference spend, and "just make the model shorter" is a cheaper intervention than another round of architecture surgery.