AI coding agents ace tidy, single-file bug fixes, but a new benchmark suggests they fall apart once a bug crosses process and node boundaries.
Researchers built DDBench, a set of 60 historical bugs pulled from 13 open-source distributed systems and split into three difficulty tiers. Each bug was tested under two conditions: agents got only the bug's symptom and the repository, or they additionally got a bounded debugging context of logs, traces, runtime state, and investigation notes. Across ten LLMs, pass rates on the hardest cases swung by 61 percentage points, and statistical tests could tell nine of fifteen top model pairings apart. Handing agents that extra debugging context raised the overall pass rate by 18.1 percentage points, but weaker models mostly solved more bugs while stronger ones mostly got faster - and even accurate context sometimes steered models toward the wrong fix.
Frontier models have been clustering in the high-70s on SWE-bench Verified, the benchmark most labs cite as proof agents are closing in on human-level debugging. DDBench's results suggest that number describes a narrower skill than the industry's marketing implies: distributed bugs, the kind that actually take down production systems, expose a reasoning gap single-process tests never surface.
Feeding agents more logs helps, but it is not a substitute for judgment - someone still has to decide which traces are worth reading, and DDBench shows models can be misled even when the evidence they're given is accurate.