Physics AI models look impressive until you move the goalposts.
Researchers built a benchmark testing five physics foundation model architectures across 8 physical dynamics, 3 training-data mixtures, and 25 test regimes — covering in-distribution, shifted, and fully out-of-distribution settings. With four model variants per architecture, they logged 60,000 measurements. The verdict: these models behave as "conditional" generalists, not universal ones. Performance depends heavily on the physical regime, temporal scale, initial conditions, pretraining, model size, and architecture.
This matters because physics foundation models are increasingly pitched as general-purpose simulators for science and engineering. If they only work well under the conditions they were trained on, their practical value narrows considerably — and any benchmark that reports a single average score is hiding that fragility rather than measuring it.
Neither scaling up model size nor expanding training data reliably fixed the problem. The researchers argue the field needs new learning mechanisms that can transfer physical knowledge across regimes, not just more of the same. That is a pointed critique of the current scaling playbook — and one the labs pushing these models will need to answer.