A new training technique stops small AI agents from spiraling into compounding errors when they are trained by a bigger model.
On-policy distillation trains a smaller student language model by having a stronger teacher model grade the student's own generated responses token by token, and it has worked well on tasks like math problems. Multi-turn agentic tasks are harder: each action a student takes changes what it sees next, so an early mistake snowballs and the student's trajectory can end up far from anything the teacher would produce. Once that happens, the teacher's feedback stops being useful and can even make training worse. The new method, called STI-OPD, has the teacher step in and replace the student's proposed action with its own, triggered by a stochastic process that measures the policy gap with KL divergence, turns it into a probability, and randomly samples whether to intervene rather than acting only past a fixed threshold or on a preset schedule; a companion Importance-Weighted Reverse KL objective then corrects for the mismatch between teacher-written and student-generated text so the training signal stays accurate.
This targets a real bottleneck in how agentic AI gets built right now: as labs train models to use tools, browse the web, or carry out long multi-step tasks, the gap between a rollout and a teacher's intended path grows with every turn, and naive distillation methods either over-correct or ignore the drift entirely. STI-OPD's results back up the fix: it beat the strongest prior on-policy distillation baseline on every benchmark and student model size the researchers tested, covering both tool-use reasoning and long-horizon interaction tasks.
It is a plumbing-level improvement, not a new capability, but plumbing like this is what decides whether an AI agent trained on a teacher's examples can actually hold up over twenty steps instead of just the first two.