Three of the most capable AI models available today failed a physics test designed to be impossible to cram for.
Researchers built a four-stage diagnostic — induction, formulation, prediction, and review — to test whether large language models can reason inside physics frameworks they have never seen. The three frameworks include a counterfactual world where force equals mass times velocity instead of acceleration, a reconstruction of Aristotelian mechanics, and a four-domain invented system called Decay World. Composite pass rates across the three frameworks were 6/15, 6/15, and 0/15 respectively. (The paper names specific model versions tied to those figures; because those identifiers could not be independently verified against publicly documented model lines at publication time, this article withholds them.) The study released all prompts, responses, verdicts, and audit records.
The sharpest finding is a qualitative-versus-quantitative split: models almost always got the direction of a change right in Decay World but frequently computed the wrong ratio by reverting to standard-physics formulas. That pattern suggests these systems are doing something closer to plausible extrapolation than first-principles reasoning — they can sense "more" or "less" but reach for memorized equations when actual calculation is required.
The methodology findings are almost as damaging as the physics scores. LLM-judge reliability did not transfer across frameworks, undermining a common shortcut in AI evaluation pipelines, and self-review was weak in every test: models wrongly reported no earlier error in at least two-thirds of trials that actually contained one.
Most LLM benchmarks test whether a model has seen a problem before; this one is specifically designed to make that impossible, which makes a 0/15 composite harder to explain away as a dataset contamination problem.