A new attack technique fools the safety filters built into Vision Transformer models by planting a decoy that draws their attention away from the actual threat.
Researchers published a paper describing "adversarial decoys" — optimized image patches designed not to cause misclassification themselves, but to hijack the attention-ranking mechanism that many test-time defenses rely on. Current defenses against adversarial patches work by suppressing image regions that attract unusually high attention scores, on the assumption that those regions are the threat. The decoy exploits that assumption: it is optimized independently of the main adversarial patch, pulling attention scores toward itself so the real attack slips through unsuppressed. Experiments on ImageNet across several ViT architectures confirmed that decoys can redirect high attention away from the genuine adversarial region while preserving much of the attack's effectiveness.
The method exposes a structural flaw in a whole class of defenses, not just one product or model. Because the decoy is attack-agnostic — optimized separately from whatever underlying patch attack it is paired with — it can in principle be grafted onto any existing adversarial patch technique, widening the threat surface considerably.
Attention magnitude has been treated as a reliable proxy for adversarial relevance in ViT defenses; this paper argues that proxy is gameable by design, which is an uncomfortable finding for anyone shipping vision models in security-sensitive settings like facial-recognition or autonomous systems.