Researchers have found a way to train small language-model agents that eventually outperform the larger models they learn from.
The technique, called ATOD (Annealed Turn-aware On-policy Distillation), blends two training approaches that each have a known weakness. Imitation learning - where a small model copies a larger teacher - starts fast but plateaus once the student catches up. Reinforcement learning explores for higher rewards but is slow and inefficient early on because feedback is sparse. ATOD schedules these two methods in sequence: imitation dominates early training, then reinforcement learning gradually takes over. A second component called Turn-level Disagreement-Uncertainty Reweighting (T-DUR) identifies which steps in a long task are most worth learning from and weights them accordingly. Tested on three agent benchmarks - ALFWorld, WebShop, and Search-QA - ATOD improved average success rates by about 3 points over pure imitation and more than 23 points over a reinforcement-learning-only baseline, while edging past the teacher models themselves by roughly 2 points.
The teacher-surpassing result is the part worth noting. Most distillation research frames the teacher as a hard ceiling; cracking it, even slightly, suggests the RL phase is doing genuine exploration rather than just recovering from imitation's plateau. For teams trying to deploy capable agents without the inference cost of frontier models, a method that wrings more out of a smaller model is directly useful.
That said, the gains are measured on simulation benchmarks with relatively clean reward signals - real-world tasks, where rewards are noisier and trajectories longer, tend to punish exactly the kind of exploration RL depends on.