A new decoding trick promises to make Mixture-of-Experts AI models run noticeably faster on phones, laptops, and other memory-constrained edge hardware.
Researchers describe S2-MoE, a self-speculative decoding framework built for Mixture-of-Experts inference on edge devices. It combines three tricks: routing-aware adaptive speculative expansion to cut wasted verification steps, reuse-aware expert gating to make better use of experts already loaded in memory, and shared context to keep the draft and target models in sync. Implemented in llama.cpp, a popular open-source engine for running LLMs locally, S2-MoE reportedly hits up to 5.3x faster decoding than standard autoregressive generation, averaging around 2.0x across the models and datasets tested. Code is posted on GitHub for anyone who wants to check the claims themselves.
The pitch here is narrow but real: Mixture-of-Experts models only activate a fraction of their parameters per token, which should make them cheaper to run, but constantly swapping experts in and out of an edge device's limited memory eats those savings. Bolting speculative decoding on top usually makes things worse, not better, because verifying guessed tokens against the wrong experts wastes cycles. S2-MoE is aimed squarely at that mismatch.
It is worth remembering the 5.3x figure is a best case, not a typical one - the average gain lands closer to 2x, and that number comes from the authors' own benchmarks rather than independent testing on a range of real consumer hardware.