A new scheduler lets AI labs squeeze multiple expert models into one smaller model faster, by only spending compute where a model is still improving.
The technique, called D3-MOPD, targets multi-teacher on-policy distillation: training a single "student" model to imitate several domain-expert "teacher" models at once. The standard approach fixes how much training data comes from each teacher's domain before training starts. Researchers found that wastes compute, since some domains converge quickly and plateau while others keep improving throughout the run. D3-MOPD instead watches each domain's existing convergence signal in the background and shifts the training mix toward domains still making gains, without altering the core training loop.
On a 35-billion-parameter student model learning from four domain-expert teachers, the scheduler closed 97% of the performance gap to those teachers, versus 63% for the standard fixed-mix approach, while reaching the same peak performance in roughly a third of the rollout steps. It even beat the specialist teachers outright on three of seven benchmarks. Training compute is the main cost of distillation, which is how many teams turn expensive frontier models into cheaper ones they can actually run, so a free efficiency gain here compounds fast.
The appeal is that it is a "zero-overhead" add-on, not a rebuild: no changes to the training loop, just smarter bookkeeping. That is the kind of unglamorous fix that tends to actually get adopted. Still, this is one model family and four teacher domains, so how well the gains hold up on smaller students or messier domain mixes is an open question.