A new benchmark wants to end the guessing game around which AI models actually find Java vulnerabilities - and which ones just look like they do.
Researchers released JavaVulBench, a dataset of roughly 30,600 Java methods drawn from over 700 projects and mapped to 1,740 CVEs. Labels exist at both method and line level, and each CVE carries a publication date so evaluators can construct five different train-test splits: random, project-disjoint, temporal, deduplicated, and unseen CWE family. A unified harness wraps twelve detectors - including CodeBERT, GraphCodeBERT, UniXcoder, DeepSeek-Coder-1.3B, GPT-4o, GPT-4.1-mini, Claude Sonnet 4, and several Qwen and CodeLlama variants - so every model runs under identical conditions from a single command. The whole package, including fine-tuned checkpoints, is archived on Zenodo.
The harder-to-ignore feature is the contamination audit. Because large language models are pre-trained on public code and CVE disclosures, a model that "detects" a known vulnerability may simply be recalling it from training data. JavaVulBench ships a per-model audit that flags which test CVEs a given model might have already seen, letting researchers separate genuine detection from memorization. That distinction has been mostly hand-waved away in prior vulnerability benchmarks.
The field has no shortage of security benchmarks, but most cover C and C++ - Java's enterprise dominance makes a credible Java-specific dataset overdue. Whether the twelve reference models represent a meaningful cross-section is another question; notably absent are larger open-weight code models and any fine-tuned security-specialist variants. Still, a benchmark that at least tries to measure what it claims to measure is a better starting point than one that doesn't.