A new routing method lets Mixture-of-Experts models make smarter decisions by borrowing information from their own attention patterns, not just a single snapshot of the hidden state.
In MoE language models, a router decides which expert sub-network handles each token, typically based only on that token's current hidden state. A team of researchers built Attention-Aware Routing (AAR), which adds temporal and spectral signals pulled from a sliding window of attention weights, essentially a compressed history of what the model has been focusing on. They froze the rest of the transformer and trained only the routing parameters on OLMoE, isolating routing as the single variable that changed. AAR lifted GSM8K math benchmark scores by 3.37 percentage points over a routing-only baseline, and wrong answers got shorter while correct ones stayed the same length.
The more interesting finding is mechanistic: routing decisions at one layer measurably reshape attention behavior, specifically strengthening so-called attention sinks, at the next layer, even though nothing about attention itself was directly changed. That means routing and attention aren't separate systems bolted together, they're a coupled circuit influencing each other through the residual stream. It's a small but genuine clue about how these models actually process information, distinct from another leaderboard-chasing tweak.
The catch: applying AAR indiscriminately across every layer hurts factual retrieval, even as it helps math reasoning when placed deeper in the network. This isn't a drop-in upgrade yet. It's a scalpel, tested so far on exactly one model.