Give an AI coding agent a blank slate and it writes decent code; give it your actual architecture and it starts breaking things.
A new study on arXiv tested coding agents across 100 backend generation tasks spanning eight web frameworks, using a fixed API contract to isolate how well agents follow structural rules like architectural patterns, databases, and object-relational mappers. Researchers scored the agents with both end-to-end behavioral tests and static verifiers, then compared performance on loosely specified tasks against fully constrained ones. Average assertion pass rates dropped 27.28 points as structural requirements piled up, a pattern the authors call constraint decay. Mid-tier models held up fine in minimal frameworks like Flask but performed markedly worse in convention-heavy ones like FastAPI and Django, and the most common failures were data-layer bugs: bad query composition and ORM violations.
Most coding-agent benchmarks reward functionally correct output and ignore whether it fits a codebase's actual rules, which is backward for production work. This study suggests the gap between demo-ready AI coding and deployable AI coding is mostly about structure, not logic, and it shows up hardest in the frameworks professional teams actually use because of their conventions.
Django and FastAPI got popular precisely because their conventions are strict enough that humans mess them up too, and it turns out autocomplete-with-ambition still can't reliably hold onto them either.