A new reinforcement learning algorithm says agents can learn faster by reconsidering their own past and future moves.
Researchers publishing on arXiv (2406.03678) describe Reflective Policy Optimization, or RPO, an on-policy reinforcement learning method that builds on existing algorithms like Trust Region Policy Optimization and Proximal Policy Optimization. Instead of updating a policy purely from the current state, RPO folds in information about past and future state-action pairs, letting an agent revise its choice within the same state. The team says this introspection step provably improves policy performance with each update and narrows the space of possible solutions, which speeds up convergence. They tested RPO on two reinforcement learning benchmarks and report better sample efficiency than baseline methods, with code posted on GitHub.
Sample efficiency is the bottleneck that keeps on-policy methods like PPO expensive to run: they need fresh data for every update, which costs compute and wall-clock time. If RPO's approach holds up outside these two benchmarks, it could mean fewer training runs to reach the same performance, relevant to anyone paying for RL infrastructure, from robotics labs to game-playing agents.
The paper is a v2 replace-cross submission on arXiv (2406.03678), and the authors have not shown whether RPO holds up in the larger, messier environments where PPO and TRPO are actually deployed today. That gap is what separates a good benchmark number from a method teams actually adopt.