A new reinforcement learning technique aims to stop AI reasoning models from wasting training time re-polishing skills they have already mastered.
Most post-training for reasoning models like the ones behind AIME-style math benchmarks uses reinforcement learning that compares a batch of model outputs, or rollouts, against each other. When researchers want a model to satisfy several goals at once, such as correctness and format, the standard approach mashes those reward signals into one fixed-weight score before making that comparison. The new paper shows this can give two very different outputs the exact same score, and it keeps spending training effort on goals the model has already nailed instead of the ones still lagging. The proposed method, called SA-MRPO, scores each reward objective separately and automatically dials down its influence once a batch shows that objective is largely satisfied, shifting the training budget toward whatever still has room to improve.
That matters because reward shaping is the unglamorous plumbing behind every reasoning-focused model release this cycle, and multi-goal training is more the norm than the exception. Compared with a baseline method called GDPO, SA-MRPO improved the harder correctness objective in 12 of 15 benchmark comparisons, including a 5 percent jump on the AIME24 math test, a 3.8 percent average gain across five adaptive-reasoning benchmarks, and up to 9.2 percent on AMC23, all while keeping the easier objectives from backsliding.
The numbers are solid but not staggering, and they come from one arXiv paper measured against one baseline on a narrow slate of math and coding benchmarks, so treat this as a promising training trick, not a proven industry standard yet.