[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"branding":3,"analytics":7,"article-no-code-bug-fixes-from-llms-mostly-dont-work-study-finds":10,"sections":41},{"siteName":4,"siteTagline":5,"publisherName":4,"contactEmail":6},"The Revision","Tech news, decoded.","editor@therevision.news",{"gaMeasurementId":8,"adsenseClientId":9},"G-ZW2MV82GYR","ca-pub-8533917693782264",{"article":11},{"id":12,"slug":13,"title":14,"dek":15,"body_md":16,"tags_json":17,"published_at":18,"created_at":19,"updated_at":20,"status":21,"review_note":22,"review_notes":23,"image_url":22,"persona_id":22,"persona_name":22,"section":30,"tags":31,"sources":36,"feedback":40,"feedback_at":22,"cost_usd":40,"total_tokens":40},11035,"no-code-bug-fixes-from-llms-mostly-dont-work-study-finds","No-Code Bug Fixes From LLMs Mostly Don't Work, Study Finds","A study finds LLM no-code bug fixes resolve issues 14.6% to 49.7% of the time depending on the executor, peaking at 74.1% for the best pairing.","A new benchmark finds that AI-written no-code bug fixes, the kind that just tell you to flip a setting instead of patching code, often don't actually fix anything.\n\nResearchers built an automated pipeline that checks whether a proposed no-code fix (changing a setting, upgrading to a version where the bug is already fixed, or adjusting a workflow) actually resolves the reported bug, rather than relying on a developer to verify it by hand. They ran 322 no-code fixes generated by 12 different LLM configurations through a real browser: an executor agent carried out each fix's instructions, and a separate checker confirmed whether the bug persisted. Three executors took turns running the tests, OpenCUA-72B, Claude Sonnet 5, and Meta's Muse Glimmer. Resolution rates across all 322 fixes ranged from 14.6% to 49.7% depending on which executor ran them, and the strongest single combination, Claude Opus 4.6's fixes carried out by Claude Sonnet 5, cleared 74.1%.\n\nThe more telling number isn't the top score, it's the variance. Swapping only the executor shifted a configuration's resolution rate by 38.8% on average, and the three executors agreed on the same verdict for under half the fixes. That means a fix's reported success rate today says as much about which agent tested it as about which model wrote it, a shaky basis for routing automated fixes straight to users.\n\nEven the executors' calls only matched human judgment 66.1% to 88.1% of the time, a reminder that automated verification is a useful sanity check right now, not yet a replacement referee.","[\"llms\",\"bug-fixes\",\"ai-agents\",\"software-testing\"]","2026-10-09T04:00:00.000Z","2026-10-10T03:05:38.974Z","2026-10-10T03:05:44.288Z","published",null,[24],{"id":25,"reviewer":26,"round":27,"reason":28,"status":29},"editor-r1","editor",1,"The dek claims the best performer resolved less than half its fixes, but the body states the best combination (Claude Opus 4.6\u002FClaude Sonnet 5) cleared 74.1 percent — fix the dek to match the body's actual figures (e.g., the executor-dependent 14.6-49.7% range) instead of contradicting them.","resolved","dev-tools",[32,33,34,35],"llms","bug-fixes","ai-agents","software-testing",[37],{"name":38,"url":39},"arXiv cs.AI","https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.11963",0,{"sections":42},[43,47,51,56,61,65,69,74,79,84,87,92,97,102],{"name":44,"slug":45,"count":46,"latest_published_at":18},"AI","ai",6783,{"name":48,"slug":49,"count":50,"latest_published_at":18},"Security","security",932,{"name":52,"slug":53,"count":54,"latest_published_at":55},"Policy","policy",486,"2026-10-08T22:40:11.000Z",{"name":57,"slug":58,"count":59,"latest_published_at":60},"Deals","deals",474,"2026-10-08T22:00:00.000Z",{"name":62,"slug":63,"count":64,"latest_published_at":18},"Hardware","hardware",232,{"name":66,"slug":67,"count":68,"latest_published_at":18},"Science","science",193,{"name":70,"slug":71,"count":72,"latest_published_at":73},"Consumer Tech","consumer-tech",181,"2026-10-08T23:26:35.000Z",{"name":75,"slug":76,"count":77,"latest_published_at":78},"Startups","startups",117,"2026-10-08T16:45:00.000Z",{"name":80,"slug":81,"count":82,"latest_published_at":83},"Software","software",114,"2026-10-08T17:57:01.000Z",{"name":85,"slug":30,"count":86,"latest_published_at":18},"Dev Tools",106,{"name":88,"slug":89,"count":90,"latest_published_at":91},"General","general",66,"2026-10-09T04:46:11.000Z",{"name":93,"slug":94,"count":95,"latest_published_at":96},"Gaming","gaming",58,"2026-10-08T20:08:45.000Z",{"name":98,"slug":99,"count":100,"latest_published_at":101},"Reviews","reviews",34,"2026-10-08T14:00:22.000Z",{"name":103,"slug":104,"count":105,"latest_published_at":106},"How-To","how-to",8,"2026-10-05T09:00:00.000Z"]