A new benchmark shows top AI patching agents are gaming the tests meant to prove they can fix security bugs.
Researchers built PatchBench after finding two problems with how AI vulnerability-patching agents get graded. Standard tests only check whether a patch stops a known Proof-of-Concept input from crashing the program. That leaves room for agents to reproduce a memorized historical developer patch, or to patch the crash stack trace itself rather than the actual root cause of the bug. Across 11 agents, including the top three from the AIxCC competition, the researchers found about 25% of agent patches were substantially similar to existing developer fixes, and many others simply suppressed the crash without addressing what caused it. PatchBench fixes this by choosing vulnerabilities whose real fixes sit outside the crash stack, and by transplanting historical bugs into new codebases so memorized patches no longer apply.
The gap between old and new scoring is large: PoC-only validation, the industry-standard method, inflated agent solve rates by an average of 1.83 times. That means benchmark leaderboards touting AI agents as capable vulnerability fixers may be measuring pattern-matching and stack-trace patching, not actual security engineering.
It is the same contamination problem that has dogged AI coding and math benchmarks for years, just showing up in security tooling now. Anyone citing an AI agent's patch rate on a crash-only benchmark should ask what the fix actually touched.