A new fine-tuning trick makes mixture-of-experts AI models noticeably better at tasks in languages other than English, by fixing how they route work internally rather than just retraining on more data.
Mixture-of-experts models split their work across specialized sub-networks called experts, with a router sending each input to just a few of them; researchers studying several such models found that in the middle layers, the routing pattern for a task looks similar in English and in other languages, and diverges when the model struggles with that language. Building on that observation, the team built RA-MoE, a three-stage fine-tuning method that sorts matched English and non-English example pairs into four groups based on whether the model got each version right or wrong. It then nudges the non-English routing on the mixed-result examples toward the pattern that worked for English, matching both the amount of work sent to task-relevant experts and how that work is split among them. Tested on three MoE models, three tasks, and six target languages, RA-MoE beat both standard fine-tuning and other routing-aware methods.
Companies fine-tuning MoE language models for non-English markets typically tweak the weights and call it done, ignoring the routing layer entirely - this work suggests that is exactly where some of the lingering language gaps live. The deeper finding is that the relevant task knowledge is often already in the model for other languages; it is just being routed to the wrong experts, which points to cheaper fixes than retraining or adding parameters.
That said, this is a result on a handful of open models and six languages, not a drop-in patch for the multilingual assistant your company already shipped - and routing tricks that shine on benchmarks have a habit of getting quieter once they meet real traffic.