Small language models keep failing at reinforcement learning, and a new arXiv paper says the problem isn't size - it's plumbing.
Researchers ran fifteen combinations of small models, from 70 million to 500 million parameters, including Pythia and SmolLM2 variants, through Proximal Policy Optimization on three text corpora. They found three specific bugs behind the instability: LoRA adapter weights silently freezing in standard fine-tuning pipelines, numerical overflow in bfloat16 precision during policy updates, and reward-model errors triggering full policy collapse. The fixes were mundane rather than clever - reinitializing frozen adapters, switching to float32 during PPO updates, and adding safety checks like reward whitening and weight rollback. With those fixes in place, training converged reliably across all fifteen configurations.
The team's real claim is a capacity-headroom hypothesis: RL performance at this scale depends on whether the starting model is fluent (perplexity under 20) and whether the reward signal is actually informative, not on parameter count. That reframes small-model RL failures as an engineering problem rather than a scaling limitation - useful for anyone trying to run RL fine-tuning without a GPU cluster.
It reads like an infrastructure postmortem more than a new algorithm, and these models still lag far behind anything you'd deploy. But the checkpoints and code are public, and if the bug list holds up outside three toy corpora, it's a solid debugging checklist for the next person who hits the same wall.