Researchers have a new explanation for why reinforcement learning with verifiable rewards actually works.
A paper out of arXiv proposes a theory of training dynamics for a class of AI models that learn by solving compositional reasoning tasks. The core finding: when a model trains on a mix of easy and hard problems, it naturally develops an implicit curriculum — easier problems get solved first, and the gradients they produce make slightly harder problems tractable next. No one programs this progression in; it emerges from the math. The researchers call the well-behaved version of this dynamic a "relay regime," where the model perpetually trains at the edge of its own competence.
This matters because RLVR has driven most of the recent gains in large reasoning models — systems like those behind o3 and DeepSeek-R1 — yet the mechanism was poorly understood. Rewards based on final outcomes only tell a model whether it got the answer right, not how to get there. This theory offers a principled account of how a model can still bootstrap its way to extended, multi-step reasoning from that sparse signal alone.
The catch is in the difficulty spectrum. When the gap between easy and hard problems is smooth, training behaves nicely. When there are abrupt jumps in difficulty, training stalls in prolonged plateaus before suddenly lurching forward — a pattern the authors compare to grokking. That's a useful warning for anyone designing training datasets: the shape of your difficulty distribution may matter as much as its contents.