Researchers have a new approach to slimming down Mixture-of-Experts AI models without needing specialized training data.
Large language models built on the Mixture-of-Experts architecture route each input through only a subset of "expert" sub-networks, which keeps inference costs down. The problem: pruning those experts — removing redundant ones to shrink the model further — usually requires calibration data tailored to the target task. A new method called Generic TB-Coverage sidesteps that requirement. It profiles each expert's usefulness separately on two generic text corpora (WikiText2 and C4), then enforces a budget rule that keeps the highest-utility experts from each corpus before building the final pruning mask.
Tested on Qwen1.5-MoE-A2.7B and DeepSeek-MoE-16B-Base at three retention budgets (25%, 50%, and 75%), the method outperformed random pruning, REAP, and ExpertSparsity across six zero-shot benchmarks. Gains were sharpest at aggressive pruning rates, suggesting the cross-corpus coverage heuristic is most valuable when the model is cut hardest.
MoE models have become a preferred architecture for frontier labs — DeepSeek's cost-efficient models helped put the design on the map — so better pruning techniques matter at scale. The catch is that this research evaluates relatively modest open models; whether the gains hold for larger MoE deployments remains to be seen.