Vision-language AI models hallucinate objects partly because their attention literally drifts while they are looking.
A new paper from researchers studying multimodal large language models identifies a specific mechanism behind object hallucinations: attention distraction. In these models, the problem shows up as spatial inconsistency across attention heads and a gradual fade in how much weight the model gives image tokens as it generates text. The paper draws an analogy to divided human attention — when people split focus, their visual descriptions get fuzzier and less accurate. The same degradation, the researchers argue, is baked into how current models decode images. They also offer theoretical backing: attention dispersion measurably increases model complexity and weakens classification accuracy.
The practical upshot is a plug-in fix called AFIP — Attention-Focused Approach for Improved Image Perception — that corrects the drift through cross-head attention enrichment and a dynamic mechanism that reinforces attention to image tokens over time. Crucially, it requires no additional training, which lowers the barrier to adoption considerably; swapping in a new technique mid-inference is far cheaper than fine-tuning a model. Benchmarks across multiple models show consistent gains.
Hallucination reduction has become a crowded subfield, with approaches ranging from reinforcement learning from human feedback to contrastive decoding — this one is notable for diagnosing the visual pathway specifically rather than patching outputs after the fact.