[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"branding":3,"analytics":7,"article-new-benchmark-separates-ai-perception-errors-from-reasoning-errors":10,"sections":44},{"siteName":4,"siteTagline":5,"publisherName":4,"contactEmail":6},"The Revision","Tech news, decoded.","editor@therevision.news",{"gaMeasurementId":8,"adsenseClientId":9},"G-ZW2MV82GYR","ca-pub-8533917693782264",{"article":11},{"id":12,"slug":13,"title":14,"dek":15,"body_md":16,"tags_json":17,"published_at":18,"created_at":19,"updated_at":20,"status":21,"review_note":22,"review_notes":23,"image_url":22,"persona_id":22,"persona_name":22,"section":34,"tags":35,"sources":39,"feedback":43,"feedback_at":22,"cost_usd":43,"total_tokens":43},6547,"new-benchmark-separates-ai-perception-errors-from-reasoning-errors","New Benchmark Separates AI Perception Errors From Reasoning Errors","A new diagnostic test shows many multimodal AI models misread diagrams, not fail at the underlying physics or geometry math.","Turns out some AI models flunk physics not because they can't do the math, but because they can't read the diagram.\n\nResearchers built a five-task diagnostic that separates perception from reasoning in multimodal AI systems tackling physics and geometry problems, according to a preprint posted on arXiv (2609.18991). The setup feeds models raw images, human-written captions describing those images, and text-only versions of the same problems, then checks where accuracy breaks down. Misreading a diagram drags down performance even on problems the same model solves correctly when given only text. Accuracy climbs when raw images are swapped for accurate captions, and some models recover almost fully once the caption is corrected - evidence the bottleneck was seeing, not thinking.\n\nThat distinction matters because benchmark leaderboards usually report one accuracy number, blending two very different failure modes into a single score. The study also found that the type of reasoning error following a perception failure depends on the subject: physics mistakes turn into bad calculations, geometry mistakes turn into misapplied concepts. Knowing which one broke tells a builder whether to invest in a better vision encoder or a better solver.\n\nThe paper singles out InternS1-mini as a cautionary case: despite heavy scientific pretraining and built-in thinking steps, it landed behind every other model tested, with its reasoning traces frequently cutting off before finishing - a reminder that more scaffolding does not guarantee it gets used.","[\"ai\",\"benchmarks\",\"multimodal-ai\",\"research\"]","2026-09-17T04:00:00.000Z","2026-09-17T23:33:34.815Z","2026-09-17T23:33:46.668Z","published",null,[24,30],{"id":25,"reviewer":26,"round":27,"reason":28,"status":29},"editor-r1","editor",1,"Attribute the findings to their actual source — name it as a new arXiv preprint (e.g. 'Lost in Perception,' arXiv:2609.18991) rather than referring only to unnamed 'researchers' and 'a new study,' so readers can verify it.","resolved",{"id":31,"reviewer":26,"round":32,"reason":33,"status":29},"editor-r2",2,"The paper is now properly named and cited, but drop or verify the specific 'published September 17, 2026' date — the source material gives no exact publication day (arXiv:2609.18991 only encodes year\u002Fmonth), and citing today's date as the paper's publish date looks fabricated.","ai",[34,36,37,38],"benchmarks","multimodal-ai","research",[40],{"name":41,"url":42},"arXiv cs.AI","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.18991",0,{"sections":45},[46,50,54,59,64,68,72,77,82,86,91,96,101,106],{"name":47,"slug":34,"count":48,"latest_published_at":49},"AI",3852,"2026-09-17T08:27:09.000Z",{"name":51,"slug":52,"count":53,"latest_published_at":18},"Security","security",648,{"name":55,"slug":56,"count":57,"latest_published_at":58},"Policy","policy",338,"2026-09-11T04:00:00.000Z",{"name":60,"slug":61,"count":62,"latest_published_at":63},"Deals","deals",179,"2026-06-29T20:02:07.000Z",{"name":65,"slug":66,"count":67,"latest_published_at":18},"Hardware","hardware",154,{"name":69,"slug":70,"count":71,"latest_published_at":18},"Science","science",114,{"name":73,"slug":74,"count":75,"latest_published_at":76},"Consumer Tech","consumer-tech",99,"2026-09-09T17:27:33.000Z",{"name":78,"slug":79,"count":80,"latest_published_at":81},"Software","software",75,"2026-09-10T20:41:21.000Z",{"name":83,"slug":84,"count":85,"latest_published_at":18},"Dev Tools","dev-tools",73,{"name":87,"slug":88,"count":89,"latest_published_at":90},"Startups","startups",55,"2026-09-09T23:14:29.000Z",{"name":92,"slug":93,"count":94,"latest_published_at":95},"Gaming","gaming",43,"2026-09-10T12:18:06.000Z",{"name":97,"slug":98,"count":99,"latest_published_at":100},"General","general",41,"2026-09-08T01:57:23.000Z",{"name":102,"slug":103,"count":104,"latest_published_at":105},"Reviews","reviews",20,"2026-06-24T12:00:01.000Z",{"name":107,"slug":108,"count":109,"latest_published_at":110},"How-To","how-to",6,"2026-06-16T09:00:00.000Z"]