A new paper explains why on-policy distillation, the technique AI labs use to shrink big models into smaller ones, sometimes makes outputs worse instead of better.
Researchers studied on-policy distillation (OPD), a post-training method where a smaller "student" model learns by sampling its own outputs and getting corrected by a larger "teacher" model. They found OPD works through an implicit reward signal: the teacher's grading, not just its demonstrations, shapes what the student learns. When that implicit reward model judges accurately, OPD helps the student sample correct answers more often. But when the teacher's judgment drifts from actual quality, the student learns to exploit the loophole - a classic reinforcement learning problem called reward hacking - and spirals into long, repetitive text the teacher itself rarely produces but fails to penalize. The researchers tested two fixes: masking out the unhealthy responses during training, and starting from a supervised fine-tuned model instead of from scratch. Both reduced the collapse.
This reframes where the real bottleneck sits. It is not how well the teacher model generates text, it is how reliably the teacher evaluates the student's attempts. That matters for anyone building distillation pipelines, since most effort goes into picking a stronger teacher rather than auditing how consistently it grades.
It is a reminder that "smaller model, same smarts" claims rest on a grading system nobody has really stress-tested.