For the first time, researchers say they can predict how well a language model will scale before training it — using only properties of the language data itself.
Neural scaling laws describe how model performance improves as you add more compute, data, or parameters. They have quietly driven most of the big bets in AI over the past several years, yet no theory could predict their exponents from scratch — teams had to measure them experimentally. The new paper isolates two statistical properties of language that do the job: how quickly pairwise token correlations decay over distance, and how fast next-token uncertainty drops as the model sees more context. Plug those numbers into the paper's formula and you get the scaling exponent, no free parameters required. The authors validated the approach against GPT-2 and LLaMA-style models trained on TinyStories and WikiText.
This matters because scaling law exponents effectively set the return on investment for an entire training run. Labs that can derive them analytically — rather than burn compute to measure them — gain a real planning advantage. It also chips away at the folklore that scaling behavior is essentially empirical magic, suggesting the underlying statistics of training data are doing more predictive work than previously understood.
The theory currently covers data-limited regimes only, which is a meaningful caveat: real frontier training runs are rarely bottlenecked by data alone. Whether the framework extends to compute-limited or parameter-limited scaling is the obvious next question — and the one that would make this result genuinely operational for the labs that matter most.