A research team reports cutting LLM pretraining convergence by two-thirds using a reinforcement learning approach to data mixing.
Researchers published AC-ODM (Actor-Critic Online Data Mixing), a method that frames which training data to feed a model at each step as an RL problem. A parameterized policy continuously reweights data sources to maximize what the authors call constructive gradient interference, meaning each batch pulls the model in a coherent direction rather than a contradictory one. The method runs in two modes: one that learns a data-mixing policy on a small proxy model and transfers it to a larger target, and one that trains end-to-end from scratch without any prior model. On Pythia-1B, the approach reaches peak validation perplexity using 66% fewer training steps than leading baselines, with a 27.5% relative gain on MMLU and a 2.23x improvement in code-generation pass rates on HumanEval, at a cost of 0.4% extra wall-clock time and 2% additional memory per step.
Pretraining runs for frontier models cost tens of millions of dollars, so shaving two-thirds off convergence steps is not a rounding error. The proxy-mode transfer is the practically useful piece: it decouples data-mix optimization from model scale, meaning labs could solve the data-composition problem cheaply on a small model and carry the policy forward to a much larger run without paying the RL overhead twice.
The code is public on GitHub, which at least lets other labs verify the numbers. Whether any of this holds at trillion-parameter scale on proprietary web corpora is a question the paper politely leaves open.