A new ablation study on a small hybrid language model suggests the trendy attention mechanism may be the less essential half of the design.
Researchers built TALH, a decoder-only model that runs Multi-head Latent Attention and a custom recurrent state-space branch in parallel, then trained five variants (117-217M estimated active parameters) from scratch on the same slice of the FineWeb dataset for the same number of steps and tokens. Stripping out the state-space branch caused the largest jump in validation perplexity, to 315, while removing the attention branch barely moved it, to 239. A dense feed-forward version scored 231 and used 3.87 GB less peak training memory than a top-2 ternary mixture-of-experts variant, which scored 240. The paper, posted to arXiv on September 25, 2026 as arXiv:2609.29618 (https://arxiv.org/abs/2609.29618), is explicit that this is a single-seed run with unmatched parameter counts across variants.
If the pattern holds at larger scale, it complicates a common assumption in hybrid model design, that attention does the heavy lifting while the recurrent branch just tags along for efficiency. Here the recurrent branch mattered more, which is worth noting given how much recent architecture work, from Mamba to DeepSeek's own latent attention, treats these components as separable, swappable choices rather than parts with wildly different importance.
The authors themselves warn against reading too much into one seed and under 220M parameters, so file this under interesting anomaly rather than settled science, at least until someone reruns it with matched parameter counts and more than one random seed.