Reinforcement learning has a quiet math problem, and researchers think they found it.
A paper posted to arXiv describes "Value Flattening," a failure mode in Proximal Policy Optimization, the algorithm most large language models use for RL fine-tuning. The critic component of PPO is supposed to estimate how good a given state is, so the policy knows which moves to reinforce. The researchers found that when they measured true state values using Monte Carlo rollouts, those values swung sharply between intermediate states. The critic's own predictions, by contrast, stayed comparatively flat. They confirmed the effect in a controlled FrozenLake test and found it worsens as the state space grows, then traced the cause to an implicit variance penalty in the critic's loss function plus redundant updates from similar, temporally correlated states.
Their fix, called SP3O, is almost aggressively simple: instead of applying the value loss to every state in a response, apply it to just a few well-separated ones. Tested on Qwen3-Base with only three supervised states per response, it mitigated Value Flattening and improved policy performance consistently across model sizes and evaluation suites.
This matters because PPO is the backbone of RLHF-style training across the industry, and a critic that quietly smooths over the signal it's supposed to sharpen is the kind of bug that costs compute without ever throwing an error. If sparse supervision generalizes beyond Qwen3, it's a cheap lever labs training reasoning and agentic models could pull immediately, since it changes which states get a loss, not the underlying algorithm.
RL fine-tuning keeps producing these unglamorous but consequential fixes - GRPO dropped the critic model entirely; this one just tells it to look at less.