A new paper identifies a counterintuitive failure mode in retrieval-augmented generation: the better the retrieval, the more likely the model is to stop looking at the image.
Researchers studying large vision-language models found that when retrieved text is highly relevant to a question, it can globally suppress the model's visual attention. Image regions relevant to the question get deprioritized. The model ends up missing answers it could have reached correctly by looking at the image alone, with no retrieval at all. The paper calls this Attention Distraction. To address it, the researchers built MAD-RAG, a training-free intervention that splits the reasoning process: one pass focuses on visual grounding without retrieved context, another integrates the retrieved text, and the two are blended through attention mixing. On three standard benchmarks, MAD-RAG recovered up to 74.68% of RAG-induced failures and gained as much as 9.20 percentage points over vanilla RAG.
The prevailing assumption in RAG research has been the opposite problem: models ignoring retrieved context. This paper inverts it. Retrieved context, when good enough, can crowd out the visual signal entirely. For teams building multimodal systems on retrieval pipelines, the implication is that stronger retrieval does not automatically mean better answers.
MAD-RAG requires no retraining, which is a practical advantage. Whether benchmark gains translate to messier production environments is a separate, open question.