A new benchmark grades AI coding agents on whether they actually make a codebase healthier, not just whether their patch closes a ticket.
SWE-Prometheus tests agents across six governance dimensions - among them tests and CI, quality gates, documentation, dependency and security, and reproducible environments - using fixed repository snapshots and clean-environment checks rather than a human-written bug report. Ten AI models were run on a shared 22-repository subset, where mean Normalized Governance Improvement ranged from 0.0568 to 0.5760 and the rate of models breaking existing behavior ranged from 0% to 23%. Comparing the two top-scoring systems, Kimi-K3 pulled ahead once broken-behavior cases were counted, even though the two looked similar on valid-only comparisons. Separately, on a frozen ten-repository batch, the researchers tested a repository-blind template - a fixed, non-adaptive baseline, not one of the ten evaluated models - which scored a mean NGI of 0.272 but improved Reproducible Environment and Dependency & Security in zero of those repositories; its gains came entirely from Tests & CI, Quality Gates, and Documentation.
That baseline result is the useful part. It shows a benchmark can be fooled into rewarding a repo for looking more governed - more tests, more CI badges, more docs - without ever touching the unglamorous, harder-to-fake work of pinning dependencies or making builds reproducible. Most existing repo-level benchmarks only check whether a patch resolves a known issue; SWE-Prometheus is built to catch that kind of surface-only cleanup instead of rewarding it.
For context, doing nothing at all - the benchmark's no-op condition - scored a median NGI of zero, which is the bar every model and every template has to clear before any of these numbers mean much.