When a long-running AI agent fails partway through a task, figuring out which step actually caused the mess is harder than it sounds.
A new benchmark called LongRCA Bench tests exactly that. It compiles 1,140 failed agent trajectories across five domains, with human-labeled answers for which role was responsible and which step was the real root cause. These are not synthetic bugs planted for the test. They are genuine failures, with a median trajectory length of 145 steps. The best existing baseline method identifies the exact failing step correctly only 13.2% of the time. The paper's own method, called RCTA, does better by retrieving candidate error steps from summarized segments and tracing them back through handoff instructions, reaching 51.1% accuracy at identifying the responsible role and 24.1% at pinpointing the exact root-cause step.
That gap matters because "who's responsible" and "what exactly went wrong" are different questions, and most tools conflate them. Knowing an agent's planning module was involved in a failure is not the same as knowing which specific decision, 80 steps into a 145-step run, is where things went sideways. As companies push agents toward longer, more autonomous workflows, this diagnostic blind spot becomes an operational cost, not just an academic curiosity.
A quarter of failures correctly localized is progress over 13%, but it also means three out of four times, a developer debugging a broken agent is still doing real detective work by hand.