A new benchmark stops asking whether an AI can write code and starts asking whether it can build a working AI agent.
The benchmark, called tau-tau-bench, hands a developer agent the same starting point a real client engagement would: a business's actual records, a client who holds the requirements, a production API operations must run through, an inherited codebase, and limits on serving cost and model choice. From there, the agent has to deliver a complete customer-service agent, which is graded by deploying it against simulated users it has never seen. Across 53 tasks spanning four business domains, the paper reports its best-performing setup - which it identifies as a model called "Claude Opus 5" running inside Claude Code - passed just 23.9 percent of those evaluation runs. An expert-authored reference solution, used as a ceiling for comparison, scored 82.2 percent.
That gap matters because the job of building agents is quietly being handed to other agents, and until now there has been no real way to measure whether that delegation actually works end to end. The paper's failure analysis reads like a review of a mediocre human contractor: shallow queries instead of real understanding of the records, almost no communication with the client, and a first draft shipped as the final design.
Writing code was never the hard part - understanding what a client actually needs, and checking your work against it, still is.