A new monograph argues most AI coding agent failures aren't the model's fault - they're the system around it.
Researchers reviewed 164 academic papers, 100 practitioner reports, 29 benchmark records, and 17 case studies of deployed coding agents to build a framework for evaluating and operating them. The result is a catalog of 206 reliability findings: 193 are gated practices backed by evidence, with 56 of those developed in real depth, covering execution environments, memory and state management, permissions, and review interfaces. The remaining 13 are labeled research leads - open questions the authors flag as unresolved and not yet backed by evidence. The monograph also ships measurements from real agent deployments, runnable evaluation protocols, and five reusable agent skills mapped to their supporting evidence.
The core claim is that judging a coding agent by model benchmarks alone misses where things actually break. A fix at one layer, like better retrieval, doesn't reliably show up in end-to-end task success, and the report treats evaluation as a dependency chain where a weak link anywhere can invalidate the whole result.
It reads less like a leaderboard and more like an audit checklist - useful if you're the one debugging why your agent's harness silently swallowed a permission error, less useful if you just wanted one number to compare tools.