A new paper offers a mechanistic reason why the Muon optimizer outperforms standard alternatives when training large neural networks.
Muon has gained traction in the research community as a strong optimizer, but the reasons for its effectiveness have remained contested. A paper posted to arXiv proposes a cleaner explanation: Muon acts as an implicit residual connection during training. The key move is orthogonalizing the weight update, which sacrifices some immediate alignment with the local gradient in exchange for better-preserved representations that downstream layers can more easily use. The authors test this framing in controlled linear optimization settings and find that Muon learns representations more slowly but more usefully.
This matters because optimizer theory tends to lag optimizer practice — researchers ship something that works before they know why it works, and the post-hoc explanations often conflict. A mechanistic account that ties Muon's behavior to the well-understood concept of residual connections gives practitioners a principled basis for when to reach for it, and gives researchers a design lens for building the next one. The residual connection is one of the most durable ideas in deep learning; if Muon is implicitly doing that work during optimization, the performance gains are less surprising.
Skeptical note: this is a theoretical framing tested in linear settings, not a benchmark showing Muon beats Adam at scale because of residual dynamics — that causal link is still an open question.