A new paper argues that picking a machine learning optimizer is a system-level design problem, not a hyperparameter search.
Researchers released OmniOpt, a survey and benchmarking framework covering more than 100 optimizer methods used in large-scale model training. The project treats every optimizer update as a pass through a five-stage meta-pipeline — most existing methods engage only one or two of those stages. OmniOpt also introduces norm-constrained linear minimization oracles (LMOs) as a common geometric lens that unifies optimizers across otherwise incompatible families. Those two structural views feed a dual-dimension taxonomy that classifies each method by its update mechanism and by the training objectives it targets.
With more than 100 optimizers in circulation and no shared vocabulary for comparing them, practitioners have largely been guessing. OmniOpt's cross-domain benchmark runs representative optimizers across language model pretraining, image classification, and other training regimes, then maps their trade-offs against explicit mechanism and objective criteria — giving researchers something closer to a decision procedure than a literature review.
The benchmark framing is useful, but the real test is whether the taxonomy survives contact with next year's methods. The field has a long history of producing optimizers that resist clean categorization.