Turns out some AI models flunk physics not because they can't do the math, but because they can't read the diagram.
Researchers built a five-task diagnostic that separates perception from reasoning in multimodal AI systems tackling physics and geometry problems, according to a preprint posted on arXiv (2609.18991). The setup feeds models raw images, human-written captions describing those images, and text-only versions of the same problems, then checks where accuracy breaks down. Misreading a diagram drags down performance even on problems the same model solves correctly when given only text. Accuracy climbs when raw images are swapped for accurate captions, and some models recover almost fully once the caption is corrected - evidence the bottleneck was seeing, not thinking.
That distinction matters because benchmark leaderboards usually report one accuracy number, blending two very different failure modes into a single score. The study also found that the type of reasoning error following a perception failure depends on the subject: physics mistakes turn into bad calculations, geometry mistakes turn into misapplied concepts. Knowing which one broke tells a builder whether to invest in a better vision encoder or a better solver.
The paper singles out InternS1-mini as a cautionary case: despite heavy scientific pretraining and built-in thinking steps, it landed behind every other model tested, with its reasoning traces frequently cutting off before finishing - a reminder that more scaffolding does not guarantee it gets used.