A new benchmark shows reinforcement learning agents that train without resets often get permanently stuck the moment a real-world action can't be undone.
Researchers built REVERSAL-BENCH, a benchmark with a tunable irreversibility parameter (rho, from 0 to 1) and a "reset oracle" that verifies whether a state can still be recovered, tested across eight manipulation tasks in five physics engines. They ran standard actor-critic algorithms, safe RL methods, and specialized reset-free frameworks built specifically to train without human intervention. As irreversibility increased, the reset-free agents got trapped in states they couldn't escape, while agents trained the old-fashioned way, with periodic resets, kept learning steadily. By comparing against geometrically identical but reversible versions of the same tasks, the team confirmed the failure came from irreversibility itself, not just harder obstacles.
Reset-free RL is the mechanism that's supposed to let robots train continuously in the real world without a person resetting the scene between attempts, the whole pitch behind robots learning on the job in warehouses or kitchens. This benchmark shows that pitch breaks down exactly where real environments differ from simulation: knock something off a table or spill it, and you can't undo it, so an agent that assumed it always could just sits there, frozen. A safety shield the researchers tested could predict an unrecoverable state accurately, but it only prevented the failure when the agent had a physical way to steer clear of the trap.
Worth remembering next time a demo shows a robot learning "on the fly" in an unstructured room: the moment something breaks that can't be put back, the learning stops too.