A new training method aims to fix a quiet reliability problem in one of AI's favorite cost-cutting tricks.
Mixture-of-Experts (MoE) models save computing power by routing each token to only a few specialized expert subnetworks instead of running the whole model every time. The researchers found that when routing is imperfect, tokens can land on experts that were never trained on data like them, quietly degrading answers even when traffic across experts looks balanced. Their fix, Distributionally Robust MoE Training (DRMoET), reworks the training objective to target those worst-case routing outcomes instead of just evening out token counts, without adding computation at inference time. Tested on the FLAME-MoE recipe at 746 million and 10.3 billion parameters over up to 67 billion training tokens, it lifted the average score across seven benchmark tasks from 0.6625 to 0.6767 at the larger scale, beating a widely used auxiliary-loss-free balancing method that scored 0.6431.
That gap matters because sparse MoE designs now underpin many of the largest language models, where routing mistakes stay invisible until they surface as inconsistent answers in production. DRMoET also cut performance loss by 4.3% when researchers deliberately forced tokens to the wrong experts, suggesting the approach helps even when routing fails outright, not just when it is merely uneven.
Balanced traffic and reliable traffic turn out to be two different goals, and most MoE training to date has only chased the first one.