A new benchmark pits LLM agents against 20 real security vulnerabilities, and the best performer fixes only half.
The project, CVE-Bench, drew its test cases from 18 Python projects including Pillow, GitPython, yt-dlp, and urllib3. Five agents — three from OpenAI and two from poolside — were each assigned a vulnerability to patch inside a sandbox, then scored against security tests drawn from the maintainers' own fixes. Across 300 runs and three prompt styles (full advisory, locate, and diagnose), the top solve rate was 50%. The remaining half is the interesting part: some of those failed fixes pass every regression test while leaving the vulnerability intact.
That false-pass result isn't a footnote. A tool that turns CI green while a CVE survives doesn't just fail — it erases the signal that something is wrong, which may be harder to recover from than an obvious miss. The cost picture adds another reason for skepticism: gpt-5.5 ran at 12 times the price of gpt-5.4-mini and produced statistically similar results, suggesting the premium-model reflex in security tooling isn't backed up here.
The author's power analysis puts the task count needed to detect a meaningful within-family performance gap at roughly 700 — most published LLM security benchmarks don't get close, so their leaderboards may be telling us less than they appear to.