Reinforcement learning on big AI models is slow and costly — researchers may have found a shortcut.
A team of researchers has published a technique called Direct On-Policy Distillation (Direct-OPD) that sidesteps one of post-training's biggest headaches. Instead of running full reinforcement learning on a large target model — which requires generating thousands of sample outputs, called rollouts, at significant compute cost — Direct-OPD runs RL on a smaller, cheaper model first. It then extracts not the small model's final policy, but the change the RL training induced in it, and applies that signal to the stronger model. The key insight is treating the before-and-after checkpoint pair as an implicit reward: whatever actions RL made the weak model favor or avoid gets translated into guidance for the larger student model operating in its own context.
This matters because post-training has quietly become one of AI scaling's biggest cost centers. As base models grow, repeating a full RL run for each new version is increasingly impractical — Direct-OPD offers a way to amortize that cost across model sizes. The method also sidesteps the need to train a separate reward model, which is its own expensive and error-prone step in standard RLHF pipelines.
The results are specific enough to take seriously: Direct-OPD lifted Qwen3-1.7B's score on the AIME 2024 math benchmark from 48.3% to 62.4% in four hours on eight A100 GPUs. That's a meaningful jump on a hard benchmark, achieved without running RL on the target model at all. Whether it holds at larger scales — say, a 70B model learning from a 7B teacher — remains to be seen, but the sequential composability the authors demonstrate suggests the approach has room to grow.