Turns out the 'attention-free' alternative to transformers still does something a lot like attention.
A new arXiv paper models selective state space models (SSMs) - the architecture behind Mamba - as a dynamical system, tracking how tokens evolve layer by layer. The authors prove mathematically that the recurrence inside SSMs pushes tokens toward consensus, clustering them toward a single shared direction, the same collapsing behavior long documented in transformer attention. Using stability arguments borrowed from control theory, they show this consensus point stays stable even when the model's weights change over time, a case earlier work didn't cover. Testing on a pretrained Mamba-2 model, they trace the effect to the output gate, the component that keeps tokens from fully collapsing into that one direction.
SSMs were pitched as a genuinely different, cheaper way to process sequences, sidestepping the quadratic cost of attention. This result suggests the resemblance goes deeper than shared benchmarks: both mechanisms are doing the same underlying job of pulling tokens toward agreement, just through different math. That matters for anyone choosing between the two, since failure modes known from attention, like over-smoothed, collapsed representations, may show up in SSMs too, just controlled by a different knob.
The evidence here comes from a single pretrained model, so 'the output gate saves us' is a promising lead, not a settled law of the architecture.