A new paper shows vision language models can be tricked into seeing something that isn't there, using image tweaks invisible to humans.
Researchers describe a gradient-based method for generating adversarial images that fool VLMs, and it skips the usual expensive step of optimizing across a model's entire architecture. Instead, the attack works only on the vision encoder, the part of the model that turns pixels into something the language side can reason about. That narrower target cuts the computational cost significantly while still working well. The team tested it on four open-source models, Qwen2.5-VL, Granite-Vision, FastVLM, and Phi-3.5-Vision, in two modes: untargeted attacks that just scramble a model's read on an image, and targeted attacks that force a specific, unrelated description.
VLMs are creeping into things like content moderation, autonomous driving perception, and medical image triage, places where a wrong read has real consequences. If small, invisible pixel changes can flip a model's answer, and doing so is cheap enough to skip touching the whole model, that lowers the bar for anyone wanting to abuse these systems. The fact that the attack only needs the vision encoder is the real story here: an attacker doesn't need to understand or access the language model at all.
Image classifiers have had this problem for a decade; VLMs just inherited it, with extra steps.