A new paper argues that the standard way researchers train AI models to reason step-by-step has three built-in flaws — and offers a small middleware layer to fix all three.
Group Relative Policy Optimization, or GRPO, is the default reinforcement learning setup for teaching large language models to work through problems one step at a time. Researchers often layer a "process reward model" on top of it to give the model denser feedback than a simple right-or-wrong score at the end. The new paper identifies three problems that emerge when you do that: different feedback signals bleed into each other during a normalization step; the granularity of the feedback does not match the granularity of the decisions being graded; and the way GRPO accumulates scores over a sequence either inflates responses toward verbosity or cuts exploration short, depending on the signal's direction. The proposed fix, called PASS (Process Advantage Signal Shaping), addresses each issue with a distinct mechanism — independent normalization per signal stream, a chunking method that groups steps by value similarity, and a length-normalization step that converts cumulative scores into a per-step density.
The practical payoff is a consistent improvement in pass@1 accuracy — the rate at which a model gets the right answer on its first try — across math reasoning and multi-hop question answering benchmarks. That matters because pass@1 is the metric that reflects real-world single-shot use, not best-of-N sampling under favorable test conditions.
Process supervision is increasingly where the frontier labs are competing, so a reusable middleware that plugs into existing GRPO pipelines without requiring a new training recipe is a genuinely useful artifact — assuming the gains hold outside the paper's two test domains, which is the part worth watching.