The defenses protecting AI chatbots from prompt injection attacks are weaker than their near-zero failure rates suggest.
Researchers built an auditing framework that stress-tests fine-tuning-based prompt injection defenses before they ship, instead of waiting for attackers to invent the next state-of-the-art exploit. Their trick: reuse intermediate checkpoints saved during a defense's own fine-tuning process as stepping stones for optimization-based attacks like GCG and BEAST. Real-world attackers never get access to those checkpoints, but the vulnerabilities they expose turned out to be genuine and exploitable anyway. Applied to defenses that normally report attack success rates near zero, the method pushed that figure as high as 96 percent.
Most defenses get graded against whatever attacks exist at publication time, not the attacks that show up six months later. That gap matters because these models increasingly sit between sensitive personal data and untrusted web content, emails, and documents - exactly the surface prompt injection exploits. A defense that looks airtight in a paper can still get walked through once someone points a stronger optimizer at it.
"Near-zero attack success" has become the AI security equivalent of "bank-grade encryption" - reassuring-sounding, and rarely paired with the adversary model that would make it mean anything.