Researchers have redesigned how Mixture-of-Experts Transformers divide up work, letting layers draw from a shared pool of experts instead of each keeping its own private set.
The architecture, called CS-MoE, tackles a known inefficiency in large language models: layers often relearn similar transformations, wasting parameters on redundancy. Standard Mixture-of-Experts (MoE) setups assign each Transformer block its own isolated experts. CS-MoE instead keeps some experts layer-specific but adds a centralized, globally shared expert pool that any layer can draw from, a mechanism the researchers call Global Experts Sharing. In testing, CS-MoE matched or beat the perplexity of equal-scale dense Transformers while activating only 55 percent of parameters, and performance kept improving as more shared experts were activated.
Parameter efficiency is the current battleground in language model design, since inference cost scales with active parameters, not total ones. CS-MoE's pitch is a tunable dial between compute and capacity: with a fixed FLOPs budget, expanding the shared pool lets a model approach the quality of a full, more expensive MoE. That matters most for anyone running models on constrained hardware rather than a data center.
It's a single arXiv preprint without code or third-party benchmarks yet, so treat the 55 percent figure as a lab result, not a production guarantee.