[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"branding":3,"analytics":7,"article-new-benchmark-shows-ai-agents-struggling-to-build-agents":10,"sections":40},{"siteName":4,"siteTagline":5,"publisherName":4,"contactEmail":6},"The Revision","Tech news, decoded.","editor@therevision.news",{"gaMeasurementId":8,"adsenseClientId":9},"G-ZW2MV82GYR","ca-pub-8533917693782264",{"article":11},{"id":12,"slug":13,"title":14,"dek":15,"body_md":16,"tags_json":17,"published_at":18,"created_at":19,"updated_at":20,"status":21,"review_note":22,"review_notes":23,"image_url":22,"persona_id":22,"persona_name":22,"section":30,"tags":31,"sources":35,"feedback":39,"feedback_at":22,"cost_usd":39,"total_tokens":39},6125,"new-benchmark-shows-ai-agents-struggling-to-build-agents","New Benchmark Shows AI Agents Struggling to Build Agents","A new benchmark has AI systems build a full customer-service agent from scratch, and even top setups pass fewer than a quarter of test cases.","A new benchmark stops asking whether an AI can write code and starts asking whether it can build a working AI agent.\n\nThe benchmark, called tau-tau-bench, hands a developer agent the same starting point a real client engagement would: a business's actual records, a client who holds the requirements, a production API operations must run through, an inherited codebase, and limits on serving cost and model choice. From there, the agent has to deliver a complete customer-service agent, which is graded by deploying it against simulated users it has never seen. Across 53 tasks spanning four business domains, the paper reports its best-performing setup - which it identifies as a model called \"Claude Opus 5\" running inside Claude Code - passed just 23.9 percent of those evaluation runs. An expert-authored reference solution, used as a ceiling for comparison, scored 82.2 percent.\n\nThat gap matters because the job of building agents is quietly being handed to other agents, and until now there has been no real way to measure whether that delegation actually works end to end. The paper's failure analysis reads like a review of a mediocre human contractor: shallow queries instead of real understanding of the records, almost no communication with the client, and a first draft shipped as the final design.\n\nWriting code was never the hard part - understanding what a client actually needs, and checking your work against it, still is.","[\"ai\",\"ai-agents\",\"benchmarks\",\"dev-tools\"]","2026-09-07T04:00:00.000Z","2026-09-07T05:14:31.392Z","2026-09-07T05:14:43.299Z","published",null,[24],{"id":25,"reviewer":26,"round":27,"reason":28,"status":29},"editor-r1","editor",1,"The piece states as fact that 'Claude Opus 5' was the model tested, but no such model exists in Anthropic's current lineup (Opus 4.8, Sonnet 4.6, Haiku 4.5, Fable 5) — verify the model name\u002Fversion against the vendor's actual lineup or hedge\u002Fattribute the claim to the paper rather than stating it as a bare fact before republishing.","resolved","ai",[30,32,33,34],"ai-agents","benchmarks","dev-tools",[36],{"name":37,"url":38},"arXiv cs.AI","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.04611",0,{"sections":41},[42,45,49,54,59,64,69,74,79,83,88,93,98,103],{"name":43,"slug":30,"count":44,"latest_published_at":18},"AI",3398,{"name":46,"slug":47,"count":48,"latest_published_at":18},"Security","security",568,{"name":50,"slug":51,"count":52,"latest_published_at":53},"Policy","policy",310,"2026-09-06T15:34:34.000Z",{"name":55,"slug":56,"count":57,"latest_published_at":58},"Deals","deals",179,"2026-06-29T20:02:07.000Z",{"name":60,"slug":61,"count":62,"latest_published_at":63},"Hardware","hardware",152,"2026-09-03T09:26:48.000Z",{"name":65,"slug":66,"count":67,"latest_published_at":68},"Consumer Tech","consumer-tech",97,"2026-09-04T15:29:18.000Z",{"name":70,"slug":71,"count":72,"latest_published_at":73},"Science","science",96,"2026-09-03T22:30:00.000Z",{"name":75,"slug":76,"count":77,"latest_published_at":78},"Software","software",73,"2026-08-18T07:51:50.000Z",{"name":80,"slug":34,"count":81,"latest_published_at":82},"Dev Tools",69,"2026-08-18T04:00:00.000Z",{"name":84,"slug":85,"count":86,"latest_published_at":87},"Startups","startups",54,"2026-09-04T23:36:14.000Z",{"name":89,"slug":90,"count":91,"latest_published_at":92},"Gaming","gaming",41,"2026-07-09T04:00:00.000Z",{"name":94,"slug":95,"count":96,"latest_published_at":97},"General","general",39,"2026-09-05T16:13:57.000Z",{"name":99,"slug":100,"count":101,"latest_published_at":102},"Reviews","reviews",20,"2026-06-24T12:00:01.000Z",{"name":104,"slug":105,"count":106,"latest_published_at":107},"How-To","how-to",6,"2026-06-16T09:00:00.000Z"]