A new reinforcement learning technique is quietly reshaping how labs fine-tune large language models for agentic work — and it already powers a 750-billion-parameter model.
Researchers introduced Single-rollout Asynchronous Optimization (SAO), a training method designed to fix instability problems that plague existing asynchronous RL pipelines. Most current systems use synchronous or batch-interleaved training, which stalls when tasks run long. Asynchronous alternatives update the model as results arrive, improving throughput, but they inherit a specific headache: GRPO, the widely used optimization framework, relies on group-wise sampling that doesn't translate cleanly to async settings. SAO sidesteps this by using exactly one rollout per prompt instead of a group, then adds a strict double-sided token-level clipping strategy to keep gradient updates from going sideways.
The practical stakes are higher than the academic framing suggests. Agentic tasks — coding agents, reasoning chains, tool use — generate trajectories that can run orders of magnitude longer than a typical chat turn. A training pipeline that chokes on long rollouts is a bottleneck for the entire category of work labs are betting on right now. SAO ran stably for one thousand steps and outperformed GRPO and its variants on SWE-Bench Verified, BeyondAIME, and IMOAnswerBench, which puts it in direct comparison territory with the approaches backing frontier coding assistants.
The method is already in production: it underpins training for the open GLM-5.2 model at 750 billion total parameters. Whether the stability gains hold at even larger scales, or whether single-rollout sampling introduces its own blind spots, is the question the next set of ablations will need to answer.