[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"branding":3,"analytics":7,"article-frontier-llms-flunk-invented-physics":10,"sections":48},{"siteName":4,"siteTagline":5,"publisherName":4,"contactEmail":6},"The Revision","Tech news, decoded.","editor@therevision.news",{"gaMeasurementId":8,"adsenseClientId":9},"G-ZW2MV82GYR","ca-pub-8533917693782264",{"article":11},{"id":12,"slug":13,"title":14,"dek":15,"body_md":16,"tags_json":17,"published_at":18,"created_at":19,"updated_at":20,"status":21,"review_note":22,"review_notes":23,"image_url":22,"persona_id":22,"persona_name":22,"section":38,"tags":39,"sources":43,"feedback":47,"feedback_at":22,"cost_usd":47,"total_tokens":47},3367,"frontier-llms-flunk-invented-physics","Frontier LLMs Flunk Invented Physics","A new four-stage diagnostic finds that top AI models can sense direction in unfamiliar physics but miscalculate ratios — and rarely catch their own mistakes.","Three of the most capable AI models available today failed a physics test designed to be impossible to cram for.\n\nResearchers built a four-stage diagnostic — induction, formulation, prediction, and review — to test whether large language models can reason inside physics frameworks they have never seen. The three frameworks include a counterfactual world where force equals mass times velocity instead of acceleration, a reconstruction of Aristotelian mechanics, and a four-domain invented system called Decay World. Composite pass rates across the three frameworks were 6\u002F15, 6\u002F15, and 0\u002F15 respectively. (The paper names specific model versions tied to those figures; because those identifiers could not be independently verified against publicly documented model lines at publication time, this article withholds them.) The study released all prompts, responses, verdicts, and audit records.\n\nThe sharpest finding is a qualitative-versus-quantitative split: models almost always got the *direction* of a change right in Decay World but frequently computed the wrong *ratio* by reverting to standard-physics formulas. That pattern suggests these systems are doing something closer to plausible extrapolation than first-principles reasoning — they can sense \"more\" or \"less\" but reach for memorized equations when actual calculation is required.\n\nThe methodology findings are almost as damaging as the physics scores. LLM-judge reliability did not transfer across frameworks, undermining a common shortcut in AI evaluation pipelines, and self-review was weak in every test: models wrongly reported no earlier error in at least two-thirds of trials that actually contained one.\n\nMost LLM benchmarks test whether a model has seen a problem before; this one is specifically designed to make that impossible, which makes a 0\u002F15 composite harder to explain away as a dataset contamination problem.","[\"ai\",\"benchmarks\",\"physics\",\"llm-evaluation\"]","2026-07-02T04:00:00.000Z","2026-07-02T08:21:05.203Z","2026-07-02T08:21:07.963Z","published",null,[24,30,34],{"id":25,"reviewer":26,"round":27,"reason":28,"status":29},"editor-r1","editor",1,"The article names AI model identifiers (Claude Opus 4.7, GPT-5.5, Gemini 3.1 Pro) that cannot be verified against each vendor's publicly documented model lines and must be removed or replaced with verified identifiers before publication.","resolved",{"id":31,"reviewer":26,"round":32,"reason":33,"status":29},"editor-r2",2,"Remove the unverified model identifiers (Claude Opus 4.7, GPT-5.5, Gemini 3.1 Pro) from the body — they appear in the source paper but cannot be verified against each vendor's publicly documented model lines, and the open concern [editor-r1] remains unresolved.",{"id":35,"reviewer":26,"round":36,"reason":37,"status":29},"editor-r3",3,"The body still names three frontier models tested without identifying them — the draft omits the model identifiers (Claude Opus 4.7, GPT-5.5, Gemini 3.1 Pro) from the prose, which is good, but the source material cannot verify those identifiers against publicly documented model lines, and the draft must explicitly note that model identities are withheld or unverifiable rather than silently dropping them, since the pass-rate figures (6\u002F15, 6\u002F15, 0\u002F15) are tied to specific models in the source and","ai",[38,40,41,42],"benchmarks","physics","llm-evaluation",[44],{"name":45,"url":46},"arXiv cs.AI","https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.00276",0,{"sections":49},[50,54,59,64,69,74,79,84,89,94,99,103,108,113],{"name":51,"slug":38,"count":52,"latest_published_at":53},"AI",2590,"2026-07-16T04:00:00.000Z",{"name":55,"slug":56,"count":57,"latest_published_at":58},"Security","security",294,"2026-07-15T19:59:48.000Z",{"name":60,"slug":61,"count":62,"latest_published_at":63},"Deals","deals",179,"2026-06-29T20:02:07.000Z",{"name":65,"slug":66,"count":67,"latest_published_at":68},"Policy","policy",158,"2026-07-16T00:02:48.000Z",{"name":70,"slug":71,"count":72,"latest_published_at":73},"Hardware","hardware",122,"2026-07-14T19:46:26.000Z",{"name":75,"slug":76,"count":77,"latest_published_at":78},"Consumer Tech","consumer-tech",93,"2026-07-13T13:20:48.000Z",{"name":80,"slug":81,"count":82,"latest_published_at":83},"Software","software",70,"2026-07-13T19:52:25.000Z",{"name":85,"slug":86,"count":87,"latest_published_at":88},"Science","science",66,"2026-07-10T10:29:37.000Z",{"name":90,"slug":91,"count":92,"latest_published_at":93},"Dev Tools","dev-tools",59,"2026-07-07T04:00:00.000Z",{"name":95,"slug":96,"count":97,"latest_published_at":98},"Gaming","gaming",41,"2026-07-09T04:00:00.000Z",{"name":100,"slug":101,"count":97,"latest_published_at":102},"Startups","startups","2026-06-29T20:55:50.000Z",{"name":104,"slug":105,"count":106,"latest_published_at":107},"General","general",29,"2026-07-10T22:28:58.000Z",{"name":109,"slug":110,"count":111,"latest_published_at":112},"Reviews","reviews",20,"2026-06-24T12:00:01.000Z",{"name":114,"slug":115,"count":116,"latest_published_at":117},"How-To","how-to",6,"2026-06-16T09:00:00.000Z"]