A new training trick lets AI labs run reinforcement learning faster without quietly corrupting the model that judges progress.
Asynchronous reinforcement learning, where rollout workers keep generating data while the policy keeps updating in the background, is now the default way labs scale RL training for language models, because it keeps GPUs busy instead of idling. The catch is the critic, the component that scores how good a given step was: by the time it grades a batch of rollouts, the policy that produced them is already several updates old, so the critic learns a biased picture. Prior fixes for asynchronous LLM training patched the policy itself but left this critic bias unaddressed. Standard critic-correction tricks from classical RL do not transfer to long, multi-step agent tasks, because correcting over a short window drops the reward signal out of the calculation, while correcting the full trajectory lets the math compound exponentially with length.
That tradeoff matters because asynchronous training is not optional at LLM scale: it is the only way labs afford the compute, so a corrupted critic would quietly cap how good these systems get, regardless of tweaks elsewhere in the pipeline. The fix, called BRACE, corrects only a prefix of tokens and appends a fixed-weight estimate for everything after it, keeping the staleness correction separate from reward tracking. On BrowseComp-Plus, a benchmark for web-browsing agents, that bought a 2.4 percentage point accuracy gain over the best prior method, 2.46 times faster training steps than synchronous training, and stability through 50 rounds of staleness.
It is plumbing, not a new capability: the unglamorous kind of fix that decides whether next-generation agentic models train in days instead of weeks, not whether they get any smarter.