A 150KB image file can manipulate how a vision-language model behaves — no jailbreak text, no access to model weights required.
Researchers introduced VISOR, a technique that crafts "steering images" designed to nudge a model's internal activations toward a target behavior. Tested on LLaVA-1.5-7B, a single steering image matched the performance of activation-based steering vectors — methods that typically require deep access to a model's runtime internals — within 1-2% for positive behavioral shifts. For negative steering, VISOR pulled ahead, achieving up to 25% shifts from baseline versus the modest changes steering vectors managed. Standard system prompting, the usual knob for behavioral control, only achieved 3-4% shifts by comparison.
The finding matters because it works entirely through the visual input channel — the kind API-based and closed-source deployments expose by default. That means an attacker doesn't need a model's weights, a special API tier, or even a crafted text prompt to redirect its behavior. Defenses built around text-based filters are simply looking in the wrong place.
The researchers frame VISOR as both a tool and a warning. It maintained 99.9% accuracy on 14,000 unrelated benchmark tasks while steering was active, which means the manipulation is quiet. The AI safety community has spent years hardening text inputs; this is a reminder that the image channel has been largely unwatched.