A new training recipe promises steadier results when squeezing a big AI model's know-how into a smaller one.
On-policy distillation is how many teams turn an expensive frontier model into a cheaper one that behaves almost as well. According to a paper posted to arXiv (2609.16937), the two standard ways of doing this both have a downside: matching the teacher token by token is stable but shortsighted, while matching whole sequences captures more of the teacher's long-range behavior but gets noisier as sequences get longer. The researchers show these are two ends of the same spectrum and propose a method called gammaOPD that uses a discounted mix of both, plus a second mechanism that blends in verifiable outcome feedback so the student isn't purely copying the teacher. Tested on math and code reasoning tasks, gammaOPD beat existing distillation methods whether the student and teacher were the same size, mismatched in size, or the student was learning from multiple teachers at once.
This matters because distillation is quietly load-bearing infrastructure for the AI industry: it's how vendors ship smaller, faster, cheaper models without a full retrain from scratch. A method that stays stable across teacher-student size gaps, and that can lean on checkable outcomes in domains like math and code, is directly useful to anyone building those pipelines, not just chasing benchmark scores.
The gains so far are confined to math and code, where answers are checkable. Whether the same discounted-credit approach holds up on messier, unverifiable tasks like open-ended writing remains an open question.