An entire codebase written by AI, with every commit and error logged, just became a research subject.
The paper, "Between the Commits: Process, Error, and Claim Reliability in a Wholly AI-Authored Codebase" (arXiv:2609.29744, posted September 25, 2026), tracks the full development history of a 21,000-line Python tool built entirely by Claude, with no human-written code or tests anywhere in it. The authors built two code-provenance tracing tools and three taxonomies covering instruction intent, commit provenance, and response reliability, then applied all of it to the dataset. They found that instructions given to coding-agent CLIs skew toward comprehension, planning, and consultation rather than direct commands, and that development was mostly proactive rather than reactive. The headline numbers: 14.3% of code-generation events contained a real error later caught by the AI's own test suite, and roughly one in four to one in five of the AI's interactive responses contained a factual error.
This matters because most claims about AI coding reliability are anecdotal. Here is an actual error rate, measured against a real, working 21,000-line tool rather than a benchmark demo. It also quantifies a gap that matters for anyone letting an agent run unsupervised: the code errors get caught because the AI wrote its own tests, but the one-in-four factual error rate in conversational responses has no equivalent safety net.
Worth noting: the AI graded its own homework. The test suite that caught those code errors was also AI-authored, so the 14.3% figure measures what the system was built to catch, not necessarily every bug that's actually in there.