AI code security tools are easier to manipulate than the bugs they are supposed to catch.
Researchers tested eight large language models on code vulnerability detection and found that none of them evaluated code on its merits alone. By holding the underlying code constant and only changing surrounding context, the team triggered three well-documented cognitive biases: the halo effect (crediting reputable authors with safer code), the framing effect (shifting verdicts based on how the task was described), and the anchoring effect (letting prior analysis results steer the model's own judgment). Framing was the biggest problem, skewing model outputs by 33.2% on average across all tested models. Anchoring came in at 23.5% and halo at 18.4%. The researchers also showed a proof-of-concept black-box attack that suppressed up to 97% of vulnerabilities a model had previously flagged — without touching a single line of the vulnerable code.
This matters because LLM-based scanners are being marketed as a serious layer of security tooling, and this research suggests their verdicts are partly a function of the words wrapped around the code. Vulnerabilities requiring semantic reasoning — the subtle, hard-to-spot kind — were more susceptible than those catchable by simple pattern matching, which is precisely where automated tools are supposed to earn their keep.
Security tools that can be manipulated through metadata and framing rather than actual exploits are a different category of problem than a noisy false-positive rate — they can be weaponized.