Researchers have extended the muP hyperparameter-transfer framework to cover model upscaling, eliminating the need to retune settings at full model size.
The paper, posted to arXiv, addresses a specific pain point in how AI labs build model families. Training a large model from scratch is expensive, so recent work has explored "upscaling": initializing a bigger model from weights of a smaller trained one to accelerate convergence. The catch is that the optimizer hyperparameters — learning rate, noise schedules — may need re-tuning at the larger size, which largely defeats the cost savings. The authors introduce a width-based upscaling method that copies and perturbs weights using theoretically grounded, width-dependent scalings. They prove that under zero perturbation the upscaled model behaves identically to the base model during training, then extend muP theory to establish that hyperparameters transfer cleanly to the upscaled architecture.
The practical payoff is real: labs releasing models at multiple sizes — which is now standard practice, from Meta's Llama family to Google's Gemma lineup — spend considerable compute just retuning hyperparameters for each tier. A principled transfer method could cut that overhead without sacrificing accuracy. Empirical results on realistic datasets back the theoretical claims, though independent replication at frontier scale remains to be seen.
Scaling laws have long promised to make hyperparameter search cheaper by extrapolation, but this work argues that upscaling breaks those assumptions unless the initialization itself is designed for transfer — a caveat the wider community has mostly glossed over.