A new training method for reasoning-focused language models separates how a model explores during training from how it responds at inference time.
The paper, published on arXiv, introduces R2PO (Residual Rollout Policy Optimization). Current reinforcement learning approaches for LLMs use the same policy to generate training data and to produce final answers. The authors argue these two jobs have conflicting goals: good training needs diverse, exploratory trajectories that produce useful gradients, while good inference needs accurate, consistent outputs. R2PO adds a lightweight "Residual Rollout Head" on top of the existing policy to handle training trajectories separately, without touching the inference path. On benchmarks, the method posted a 3.4% average accuracy gain on MATH-500 and 1.3% on APPS compared to baselines.
The insight matters because the training-inference conflation is baked into most popular RL fine-tuning recipes, including the ones major labs use to build reasoning models. If the gains hold up under independent replication, it suggests current pipelines are leaving measurable performance on the table with a fixable structural assumption. The researchers also report more diverse rollouts and reduced length bias, two failure modes that often go unaddressed.
The code is public on GitHub, so the claim is at least checkable — though a 3.4% lift on a single benchmark is a long way from a production endorsement.