Researchers have published a head-to-head comparison of softmax attention and four linear-attention alternatives, plus a small routing trick that ekes out measurable gains.
The paper pits DeltaNet, Gated DeltaNet, Kimi Delta Attention, and Gated DeltaNet-2 against each other using a shared notation that makes their differences legible. Tests ran on 350M-parameter models trained for 15 billion tokens, with additional runs at 1.3B and 3B parameters for DeltaNet. The headline results: Kimi Delta Attention paired with the Muon optimizer reached the lowest validation loss in the main sweep, while a pure Gated DeltaNet stack trained with AdamW posted the highest training throughput. Hybrid stacks — mixing linear and standard attention layers — generally improved loss but at a throughput cost.
The throughput angle matters because the entire pitch for linear attention is escaping the quadratic cost of standard self-attention at long context. Any architecture that trades that speed advantage back for better loss numbers needs a compelling reason to exist. The paper also introduces Cross-Layer Value Routing, which forwards a write value between layers rather than a write error — a small change that lowers validation loss for both DeltaNet and Gated DeltaNet without the overhead of a full hybrid stack.
The authors are careful to note they did not run an inference-speed benchmark, only training throughput — a meaningful caveat when the community is as interested in fast serving as in fast training.