[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"branding":3,"analytics":7,"article-llms-can-spot-their-own-mistakes-but-not-which-fix-helps":10,"sections":35},{"siteName":4,"siteTagline":5,"publisherName":4,"contactEmail":6},"The Revision","Tech news, decoded.","editor@therevision.news",{"gaMeasurementId":8,"adsenseClientId":9},"G-ZW2MV82GYR","ca-pub-8533917693782264",{"article":11},{"id":12,"slug":13,"title":14,"dek":15,"body_md":16,"tags_json":17,"published_at":18,"created_at":19,"updated_at":20,"status":21,"review_note":22,"review_notes":23,"image_url":22,"persona_id":22,"persona_name":22,"section":24,"tags":25,"sources":30,"feedback":34,"feedback_at":22,"cost_usd":34,"total_tokens":34},5142,"llms-can-spot-their-own-mistakes-but-not-which-fix-helps","LLMs Can Spot Their Own Mistakes But Not Which Fix Helps","A new study finds LLMs can flag likely wrong answers, but struggle to predict which multi-agent protocol is worth the extra cost.","LLMs are decent at guessing when they got a math problem wrong, but bad at knowing which fix to reach for.\n\nResearchers tested four ways of solving competition-level math and science problems: answering directly, self-correcting, a planner-executor-reviewer setup, and full multi-agent deliberation. Using more than 4,000 competition math problems as the main benchmark, they had a separate probe model check each answer right after it was generated to guess whether it was wrong. That probe hit 0.8847 AUROC at spotting failures, and stayed useful for predicting whether some form of collaboration would help at all, scoring 0.7683 AUPRC. But it barely worked for picking the specific protocol, planner-executor-reviewer or full deliberation, that would actually fix the error, scoring just 0.1674 and 0.1041 AUPRC.\n\nThat gap matters because multi-agent collaboration is not free. Every extra round of deliberation costs tokens and money. Confidence-based routing can tell a system when to escalate past a first answer, but not which escalation path is worth paying for; the paper's own retrospective oracle, which always picks the best protocol in hindsight, still beats the best real-time router by 18 to 29 points across held-out tests.\n\nSo the routing layer that decides how hard an AI system should think is still mostly guesswork dressed up as strategy.","[\"llms\",\"multi-agent systems\",\"ai research\",\"benchmarks\"]","2026-08-18T04:00:00.000Z","2026-08-18T06:48:46.857Z","2026-08-18T06:48:58.708Z","published",null,[],"ai",[26,27,28,29],"llms","multi-agent systems","ai research","benchmarks",[31],{"name":32,"url":33},"arXiv cs.AI","https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.14927",0,{"sections":36},[37,41,45,50,55,60,65,70,75,79,84,89,94,99],{"name":38,"slug":24,"count":39,"latest_published_at":40},"AI",3293,"2026-08-20T04:00:00.000Z",{"name":42,"slug":43,"count":44,"latest_published_at":40},"Security","security",435,{"name":46,"slug":47,"count":48,"latest_published_at":49},"Policy","policy",210,"2026-08-19T09:32:27.000Z",{"name":51,"slug":52,"count":53,"latest_published_at":54},"Deals","deals",179,"2026-06-29T20:02:07.000Z",{"name":56,"slug":57,"count":58,"latest_published_at":59},"Hardware","hardware",140,"2026-08-19T18:25:42.000Z",{"name":61,"slug":62,"count":63,"latest_published_at":64},"Consumer Tech","consumer-tech",95,"2026-08-18T16:05:00.000Z",{"name":66,"slug":67,"count":68,"latest_published_at":69},"Science","science",90,"2026-08-19T18:41:02.000Z",{"name":71,"slug":72,"count":73,"latest_published_at":74},"Software","software",73,"2026-08-18T07:51:50.000Z",{"name":76,"slug":77,"count":78,"latest_published_at":18},"Dev Tools","dev-tools",69,{"name":80,"slug":81,"count":82,"latest_published_at":83},"Startups","startups",47,"2026-08-19T19:13:46.000Z",{"name":85,"slug":86,"count":87,"latest_published_at":88},"Gaming","gaming",41,"2026-07-09T04:00:00.000Z",{"name":90,"slug":91,"count":92,"latest_published_at":93},"General","general",33,"2026-08-18T22:18:13.000Z",{"name":95,"slug":96,"count":97,"latest_published_at":98},"Reviews","reviews",20,"2026-06-24T12:00:01.000Z",{"name":100,"slug":101,"count":102,"latest_published_at":103},"How-To","how-to",6,"2026-06-16T09:00:00.000Z"]