A research team has published TestEvo-Bench, a benchmark designed to measure how well AI coding agents handle the unglamorous job of keeping tests current with code changes.
Most existing benchmarks treat test generation as a standalone task — write a test, see if it compiles. TestEvo-Bench pulls from real commit histories across 152 open-source Java projects, packaging each task with environment configuration so results are grounded in execution metrics: pass rate, coverage, and mutation score. The current snapshot holds 746 test generation tasks and 509 test update tasks, drawn from nearly 60,000 candidate co-evolution records. Agents face two tracks: writing new tests to capture new behavior, or fixing existing tests that broke when the code changed beneath them.
The "live" part is the interesting design choice. Tasks are timestamped to the commit that produced them, and the pipeline mines new ones periodically — meaning evaluators can filter to tasks that postdate a model's training cutoff, reducing the risk that a model is effectively reciting memorized test suites rather than reasoning about code.
The four evaluated agent harnesses — built on Claude Code, Gemini CLI, and SWE-Agent — hit up to 77.5% on test generation and 74.6% on test update under generous conditions. Both numbers fall noticeably on the most recent tasks and crater under tight per-task cost limits. That gap between best-case and budget-constrained performance is the number worth watching: most real engineering teams aren't running open-ended inference budgets.