Turns out AI agents are about as good at self-diagnosis as a check-engine light with no manual.
A new benchmark called AgentChaosBench tests how well AI models can figure out why a multi-agent system failed, using its execution logs rather than just the final output. Researchers ran five applications that coordinate AI agents using the Agent-to-Agent protocol and the Model Context Protocol, then deliberately broke things: slow or unavailable tools, corrupted responses, looped delegations, and bypassed guardrails, among ten fault types in total. That produced 275 sanitized execution traces, 250 of them faulty. When they asked language models to identify the fault type from a single trace, smaller models topped out at 13.6-19.2% accuracy. Even DeepSeek-v4-pro, described as the frontier model in the test, only hit 24.8%. Asking a model to both name the fault and locate where it happened dropped accuracy to 22%, and guardrail-bypass faults were nearly impossible to catch without a reference trace to compare against.
Most agentic AI benchmarks grade whether the final answer was right, which says nothing about whether the process that produced it was sound. As companies wire multiple AI agents together to call tools and hand off tasks to each other, that blind spot gets expensive: a system can produce a plausible answer while a guardrail silently failed or a tool call looped three times behind the scenes. This benchmark is one of the first to test whether models can actually debug agentic systems from their own telemetry, not just complete tasks.
A quarter of a million dollars in Series B funding gets a press release. A benchmark showing frontier models fail at reading their own execution logs three times out of four gets a lot less airtime, and probably deserves more.