Researchers have a new way to dynamically size training batches for large language models that could cut compute costs significantly.
Most ML training runs use fixed or hand-tuned batch sizes — a process that wastes hardware cycles and requires expensive experimentation. Existing adaptive methods use a metric called gradient noise scale (GNS) to adjust batch size on the fly, but those methods assume a standard Euclidean geometry that doesn't match how modern optimizers like signSGD, Signum, or Muon actually work. A new paper derives GNS formulas that fit the actual geometry of those optimizers, then estimates the metrics efficiently using local gradients already available across distributed training nodes. On a 160 million parameter Llama model, the approach matched constant-batch validation loss while cutting training steps by up to 66% for Signum and Muon.
Batch size tuning is one of those unglamorous costs that quietly eats into ML budgets — every run you waste figuring out the right schedule is compute you don't get back. As Muon gains traction as an alternative to AdamW for large-scale training, methods that actually account for its geometry become practically relevant, not just theoretically tidy.
A two-thirds reduction in training steps on a mid-size model is promising, but the real test is whether the gains hold at the billion-parameter scale where training costs actually sting.