A new paper proposes that AI defenses stop telling attackers when they've failed.
Researchers studying agentic AI security found that the standard detect-and-block approach has a compounding problem: when a system refuses a prompt, that refusal is itself useful data. Automated attack tools — which use language models to probe, refine, and score their own attempts — can read a wall of predictable refusals as a signal to adjust. Given enough queries, the attacker's success rate trends toward one. The paper models this dynamic formally and shows it's not an edge case; it's structural.
The alternative the researchers propose is called detect-and-misdirect. Instead of blocking a flagged malicious prompt, the system returns a response that looks plausible but carries no operational value — and is specifically designed to trip up the automated judge on the attacker's end. Their proof-of-concept implementation, Contextual Misdirection via Progressive Engagement (CMPE), uses lightweight conversational responses to replace refusals without tipping off the model doing the grading. Against standard jailbreak benchmarks, CMPE cut estimated attack success rate upper bounds by up to two orders of magnitude and nearly wiped out verified successes in end-to-end runs of the PAIR and GPTFuzz attack frameworks.
The timing matters. Agentic AI systems — where models invoke tools, coordinate with other agents, and act on external data — are deploying faster than security thinking around them. Prompt injection in this context isn't just an embarrassment; it can mean a model taking real-world actions on behalf of an attacker. Misdirection shifts the economics: instead of a defender trying to build an impenetrable wall, the attacker's own automation becomes a liability.
The obvious counterpoint: sophisticated attackers can tune their judges to detect misdirection, turning this into an arms race. But making attacks more expensive is still a win — and right now, the baseline is giving attackers free feedback on every failed attempt.