A new benchmark finds that most guardrails meant to stop AI coding assistants from writing malware barely work.
Researchers built CS-Guard, the first benchmark to systematically test guardrails for code-generation security. It combines 1,000 malware-generation prompts run through 7 jailbreak techniques plus a new "fictional scenario attack" that hides malicious intent inside a fake software-development story, alongside 331 code-to-code tests covering infilling, completion, and translation. The team ran nine guardrails across seven LLMs. For text-to-code requests, jailbreaks pushed the average attack success rate to roughly 50% for many guardrails; for code-to-code tasks, success rates approached 100% on unprotected models and still ranged from 14.4% to nearly 100% with guardrails active, and the fictional-scenario attack alone hit close to 100% success across many of them.
Most AI safety marketing focuses on chatbot conversations, not the code-generation pipelines now wired into IDEs and CI systems. This benchmark suggests the guardrails bolted onto coding tools lag well behind those built for plain text, and that feeding a model existing code to translate or complete is a far easier way to smuggle malicious logic past filters than simply asking for it.
The paper's actual contribution is a public, modular benchmark rather than a solution, which is a polite way of saying nobody has a good answer yet.