A research team has proposed a more efficient way to train AI agents that must plan across many steps before reaching an answer.
The framework, called Information Gain-based Rollout Policy Optimization (IGRPO), targets a specific inefficiency in how reinforcement learning trains large language model agents on long-horizon tasks. Current methods spread their compute budget across all candidate reasoning paths more or less blindly. IGRPO instead builds a tree of possible decision paths and allocates more exploration to branches that look informative, while quietly starving branches that look like dead ends. The researchers show this adaptive budget allocation also produces a clean optimization target — a so-called teacher distribution — that guides the model's policy update, tying the exploration strategy and the training objective together in one framework.
This matters because long-horizon search — where a model has to chain together multiple retrieval or reasoning steps before arriving at a final answer — is exactly where frontier AI systems still struggle. If a smarter rollout strategy can squeeze more signal out of a fixed compute budget, it could meaningfully cut training costs or improve accuracy without requiring bigger models. IGRPO outperformed strong baselines across seven search-augmented question-answering benchmarks under identical budget constraints.
The result is a preprint, not a shipped product, and academic benchmarks rarely survive contact with messy real-world tasks — but the core idea of treating compute allocation as an information problem, rather than a scheduling afterthought, is a sensible one that the broader RL-for-LLMs field has been slow to address.