A new reinforcement learning framework argues that PPO, the algorithm behind most LLM reasoning training, is just one point in a much bigger design space, and a better point exists.
The paper introduces Generalized Proximal Policy Optimization, or GPPO, a framework for designing RL algorithms around a choice of divergence measure and a choice of estimator. The researchers show that PPO and later variants like DAPO, DPPO, and RIPO are all special cases of this two-dimensional space, not separate inventions. They back the framework with a trust-region theory that guarantees monotonic performance improvement, plus new estimator methods that give more accurate divergence estimates than standard single-sample Monte Carlo sampling. Tested on eight competition-level math and coding benchmarks, the best GPPO setup beat the widely-used GRPO baseline by up to 50 percent on math and 14 percent on coding.
That gap matters because PPO's clipping mechanism is known to clip too early and choke off exploration, and prior fixes addressed that symptom by symptom. GPPO reframes those fixes as specific coordinates on a map of divergence and estimator choices, which means teams can reason about which fix solves which problem instead of stacking heuristics.
The benchmarks are competition math and coding, not production chatbots, so how much of that 50 percent gain survives contact with messier real-world tasks is still an open question.