A new audit finds cybersecurity LLM benchmarks measure your test setup as much as the model.
Researchers audited eight cybersecurity benchmarks against 10 models - a mix of proprietary, open-weight, and cybersecurity-specialized LLMs - by treating each benchmark as a configurable measurement pipeline rather than a fixed test. They catalogued 15 systematic failure modes in how these benchmarks are scored and run. Changing a single pipeline choice, such as how an answer is parsed or graded, can swing a model's score by more than 80 percentage points and rearrange the leaderboard. Two benchmarks testing near-identical skills ranked the same models in different orders, simply because their evaluation conventions did not match. When the team built a standardized harness that fixed pipeline choices without altering what each task measures, nine of the 10 models jumped at least three rank positions on at least one benchmark.
That matters because security teams and vendors use these leaderboards to decide which model to trust with tasks like vulnerability triage or exploit analysis, treating the score as a property of the model itself. This paper says the score is at least as much a property of the grading script. A model that looks best today could just be the one whose answers happen to match a particular parser's quirks.
Benchmark gaming and irreproducibility have dogged AI leaderboards for years. This is a reminder that cybersecurity benchmarks, despite their high-stakes framing, are not exempt.