A research team has a new reinforcement learning method that outperforms standard baselines on two well-known agent benchmarks without sacrificing alignment with the true training objective.
The core problem: training LLMs on long, multi-turn tasks with sparse rewards — signals that only arrive at the end of a task — is slow and often misses successful strategies the model never sampled. Dense process rewards fix the speed problem by giving feedback at every step, but they tend to pull the model away from what it actually needs to optimize. RSPO, or Reward-Swap Policy Optimization, tries to get the best of both. It uses dense rewards to diversify the trajectories a model explores during training, then swaps back to outcome rewards when computing the actual optimization objective. The result, the authors say, is consistent gains on WebShop and ALFWorld when layered on top of GRPO, PPO, and GiGPO — three common RL algorithms.
Why it matters: most RL-for-LLM research either accepts slow convergence or accepts reward misalignment as a necessary trade-off. RSPO's swap mechanism is a relatively clean way to sidestep that trade-off, and the fact that it improves all three baselines rather than just one suggests the gains are method-level, not algorithm-specific tuning.
That said, WebShop and ALFWorld are controlled benchmarks, not production environments — real-world multi-turn agents face reward sparsity problems that are considerably messier than a simulated shopping task.