MoSAR is a new attention mechanism that lets language models decide, on their own, which words deserve a global view and which just need a glance at their neighbors.
Researchers behind the paper introduce query and key routers, placed right after positional encoding, that pick from short, medium, and global attention regimes for each input. Instead of hard-coding a sparsity pattern the way many efficient-attention schemes do, MoSAR produces a continuous, distance-dependent attention field that's learned during training. In controlled pretraining runs with matched 500-million-parameter models, MoSAR learned a narrower attention geometry without hurting language-modeling quality, and it beat dense RoPE attention on perplexity at the training context length. When tested on longer contexts than it trained on, it beat every baseline evaluated, including ALiBi, a well-regarded approach for context extrapolation.
The quadratic cost of standard self-attention is the reason long-context models get expensive fast, and most fixes so far have guessed in advance where attention can be sparse. MoSAR's pitch is that the model should figure that out itself, since which words actually relate to which other words depends on the sentence, not a fixed rule. The paper also found the learned attention pattern holds up when simplified to a discrete top-1 choice after training, hinting, but not proving, that it could run cheaper at inference too.
This is a 500M-parameter proof of concept, several orders of magnitude smaller than the models actually straining under long-context costs today, and the paper measures perplexity, not wall-clock speed or memory savings; efficient-attention ideas have a long history of looking great in academic experiments and then quietly not shipping in production models. MoSAR still has that gauntlet to run.