[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"branding":3,"analytics":7,"article-ai-agents-that-write-their-own-tools-ship-broken-code":10,"sections":34},{"siteName":4,"siteTagline":5,"publisherName":4,"contactEmail":6},"The Revision","Tech news, decoded.","editor@therevision.news",{"gaMeasurementId":8,"adsenseClientId":9},"G-ZW2MV82GYR","ca-pub-8533917693782264",{"article":11},{"id":12,"slug":13,"title":14,"dek":15,"body_md":16,"tags_json":17,"published_at":18,"created_at":19,"updated_at":20,"status":21,"review_note":22,"review_notes":23,"image_url":22,"persona_id":22,"persona_name":22,"section":24,"tags":25,"sources":29,"feedback":33,"feedback_at":22,"cost_usd":33,"total_tokens":33},4209,"ai-agents-that-write-their-own-tools-ship-broken-code","AI Agents That Write Their Own Tools Ship Broken Code","A new benchmark finds that nearly all tools synthesized by AI agents fail on inputs they never saw during the session that built them.","AI agents that generate their own software tools are quietly accumulating libraries of code that looks fine but breaks on real workloads.\n\nResearchers tested Claude Haiku 4.5 in a harness that preserved the source code of every tool an agent synthesized during a session, then replayed those tools against a held-out conformance suite the agent never saw. The results were stark: across 222 preserved tools and three protocols, 96.8% recorded a per-tool correctness score of zero. Two protocols failed every single synthesized tool; a third failed 91.7% of them. Hand-written reference implementations, run through the same suite, scored perfectly. The tools executed without errors and the in-session verifier raised no flags — they just returned wrong answers when the inputs changed.\n\nThe implication is uncomfortable for anyone building agentic systems. Task-completion rate — the standard benchmark — measures whether an agent finishes the job in front of it. It says nothing about whether the reusable code that agent produces will hold up when a future task calls it with different inputs. The researchers call this the verification-vs.-conformance gap, and they argue it is invisible to pass-rate benchmarks by design. Their new benchmark, EvolveTool-Bench, is built to make it measurable.\n\nThis gap will matter more as tool-evolving agents move from research demos to production pipelines where synthesized code gets reused, composed, and depended on across sessions — a compounding failure mode that task-completion scores will never catch.","[\"ai\",\"benchmarks\",\"agents\",\"dev-tools\"]","2026-07-07T04:00:00.000Z","2026-07-07T20:01:55.629Z","2026-07-07T20:01:58.714Z","published",null,[],"ai",[24,26,27,28],"benchmarks","agents","dev-tools",[30],{"name":31,"url":32},"arXiv cs.AI","https:\u002F\u002Farxiv.org\u002Fabs\u002F2604.00392",0,{"sections":35},[36,40,45,50,55,60,65,70,75,78,83,87,92,97],{"name":37,"slug":24,"count":38,"latest_published_at":39},"AI",2590,"2026-07-16T04:00:00.000Z",{"name":41,"slug":42,"count":43,"latest_published_at":44},"Security","security",294,"2026-07-15T19:59:48.000Z",{"name":46,"slug":47,"count":48,"latest_published_at":49},"Deals","deals",179,"2026-06-29T20:02:07.000Z",{"name":51,"slug":52,"count":53,"latest_published_at":54},"Policy","policy",158,"2026-07-16T00:02:48.000Z",{"name":56,"slug":57,"count":58,"latest_published_at":59},"Hardware","hardware",122,"2026-07-14T19:46:26.000Z",{"name":61,"slug":62,"count":63,"latest_published_at":64},"Consumer Tech","consumer-tech",93,"2026-07-13T13:20:48.000Z",{"name":66,"slug":67,"count":68,"latest_published_at":69},"Software","software",70,"2026-07-13T19:52:25.000Z",{"name":71,"slug":72,"count":73,"latest_published_at":74},"Science","science",66,"2026-07-10T10:29:37.000Z",{"name":76,"slug":28,"count":77,"latest_published_at":18},"Dev Tools",59,{"name":79,"slug":80,"count":81,"latest_published_at":82},"Gaming","gaming",41,"2026-07-09T04:00:00.000Z",{"name":84,"slug":85,"count":81,"latest_published_at":86},"Startups","startups","2026-06-29T20:55:50.000Z",{"name":88,"slug":89,"count":90,"latest_published_at":91},"General","general",29,"2026-07-10T22:28:58.000Z",{"name":93,"slug":94,"count":95,"latest_published_at":96},"Reviews","reviews",20,"2026-06-24T12:00:01.000Z",{"name":98,"slug":99,"count":100,"latest_published_at":101},"How-To","how-to",6,"2026-06-16T09:00:00.000Z"]