A new reinforcement learning method teaches AI agents to recognize when they've outgrown their own training wheels.
Researchers built RetireOPD, a technique for training multi-turn AI agents that pairs reinforcement learning with token-level supervision from a specialized teacher model. The teacher gets privileged task information up front, while a separate skill-free student learns from both environment rewards and the teacher's guidance. Rather than following a fixed distillation schedule, the student tracks how close it's getting to the teacher's performance and cuts the teacher loose once the gap stops shrinking and it hits a target fraction of the teacher's success rate, after which training continues with reinforcement learning alone. Tested on Qwen2.5 models from 1.5 billion to 7 billion parameters, the method lifted success rates on the ALFWorld household-task benchmark by 14.1 to 18.8 percent over plain RL, and WebShop shopping-simulation accuracy by 11.8 to 19 percent.
Multi-turn agent training has a stubborn problem: a whole trajectory of actions gets boiled down to one scalar reward, giving the model little to learn from step to step. Distillation from a stronger teacher was supposed to fix that, but the researchers found privileged information doesn't automatically make a teacher trustworthy, and leaning on it too long can cap how good the student becomes. RetireOPD's adaptive cutoff addresses that directly: in every setting tested, the student ended up outperforming the teacher it was trained on.
It's a preprint, not a peer-reviewed result, and ALFWorld and WebShop are simulated testbeds rather than real deployments. Still, an agent that quietly outgrows its own tutor is a cleaner story than most self-styled reasoning breakthroughs manage.