A new method called Selective Importance Sampling wants to fix one of the messier plumbing problems in modern AI training.
When labs fine-tune large language models with reinforcement learning, they use a "rollout then update" loop: the model generates responses, then learns from them. The catch is that by the time the update runs, the data is already stale — generated by a slightly older version of the model. Importance sampling is the standard statistical fix, reweighting old data to approximate what a current model would have produced. But token-level importance ratios multiply across long sequences, and the variance compounds fast enough to destabilize training. Selective Importance Sampling, or SIS, sidesteps this by borrowing from rejection sampling: it runs a token-by-token test and treats accepted tokens as effectively on-policy, assigning them a unit importance score. Only rejected tokens get the full IS correction, keeping the math honest without letting variance explode.
The practical upside is narrow but real. SIS is a drop-in modification to the policy loss function — it adds no meaningful compute overhead and slots into existing RL algorithms without redesign. The authors tested it on both dense and mixture-of-experts LLMs across math and agent benchmarks, reporting consistent gains and better stability when training data drifts off-policy.
Importance sampling variance is a known headache in RL research, not a novel discovery, but workable plug-in fixes for the LLM-scale version are scarcer than the problem's prominence would suggest — which is either an opportunity or a sign that the real solutions are still messier than a single loss tweak.