A new technique aims to make reinforcement learning for large language models more precise by being choosy about which tokens drive training.
Researchers introduced the Relative Surprisal Index (RSI), an information-theoretic metric designed to improve RL with Verifiable Rewards (RLVR) — a training approach that uses checkable correct-or-wrong signals, like math answers, to push models toward better reasoning. The core tension RSI addresses is a standing disagreement in the field: one camp says training should focus on high-entropy token positions; another warns that low-probability tokens can destabilize gradient updates. RSI couples both signals into a single measure, then uses it to filter out tokens that are either too predictable to teach anything or too erratic to learn from reliably. The resulting method, RSI Selection, keeps only tokens within a stable RSI window.
The practical gains are modest but consistent: across Qwen2.5 models at 1.5B, 3B, and 7B parameters, RSI Selection improved avg@32 accuracy on the AIME and AMC math benchmarks by 2-3 percentage points over GRPO, a widely used RLVR baseline. That margin matters because math benchmarks at this level are already competitive, and squeezing out percentage points without scaling up compute is exactly what efficiency-focused labs are hunting for.
RSI won't rewrite how frontier models are trained overnight — 2-3 points on math benchmarks is meaningful signal, not a mandate to overhaul pipelines. But if the metric generalizes beyond math to other verifiable-reward domains, it could become a quiet fixture in the RLVR toolkit.