A new survey tried to rank 21 attention mechanisms on efficiency, expressiveness, and interpretability - and found the rankings barely hold together.
Researchers reviewed a decade of attention research, from the original Bahdanau-Luong alignment method through Transformers, sparse and linear attention, IO-aware algorithms like FlashAttention, and state-space alternatives like Mamba. A single rater scored 21 methods against three criteria: efficiency, expressiveness, and interpretability. To test how stable those scores were, the authors ran a Monte Carlo simulation with 200,000 samples, allowing each score to shift by up to one point in either direction. Under that small perturbation, a method's rank moved by more than one position in 67-70% of samples on average.
That instability matters because papers routinely lean on leaderboard-style comparisons to argue one method beats another. Here, a null model built from random rank-matched data produced a similarly unstable profile, which means the scores can't reliably support fine-grained claims like "method A beats method B" - only coarse groupings, such as sorting methods into rough efficiency or expressiveness tiers.
It is the same problem benchmark culture keeps rediscovering: a table of scores looks authoritative until someone checks how much it moves when the inputs wobble by a single point.