A new benchmarking framework exposes how badly AI models stumble when asked to find exactly where a Rust vulnerability lives in real code.
Researchers introduced RustMizan, a framework built to stress-test large language model agents on Rust vulnerability analysis. Unlike earlier benchmarks that lean on small, non-compilable code snippets and simple yes-or-no classification, RustMizan includes compilable code at the crate, file, and function levels, with labels for CWE classification and precise line-level localization. It also ships a mutation framework that generates semantics-preserving code variants specifically to probe whether models are pattern-matching on training data they have already seen. Four frontier models were tested in an agentic setup with command-line access.
The results put a number on a gap that practitioners have long suspected. Binary classification — vulnerable or not — landed between 56 and 65 percent across all models, barely better than a coin flip on harder cases. Line-level localization F1 scores hovered near 20 percent, and introducing adversarial cues knocked that figure down by about 27 points. That matters because telling a security team "this file has a bug" is far less useful than pointing to the exact line.
The contamination angle is worth noting: most public vulnerability datasets are old enough that frontier models have almost certainly trained on them, inflating benchmark scores without improving real-world utility. RustMizan's mutation approach is a direct response to that problem — though whether mutation is sufficient to fully escape training-set overlap remains an open question the paper does not fully settle.