A new paper proposes a way to stop multimodal AI models from blindly searching the web every time they're asked a question.
Researchers describe mR2AG, a framework that adds two "reflection" steps to multimodal large language models used for knowledge-based visual question answering. The first step, Retrieval-Reflection, decides whether a query actually needs an external knowledge lookup before firing one off. The second, Relevance-Reflection, points the model to which parts of any retrieved material are actually useful for answering. Tested on the INFOSEEK and Encyclopedic-VQA benchmarks, the method reportedly beats both GPT-4o and prior multimodal retrieval-augmented generation systems, and it can be added to an already-trained model through fine-tuning on a purpose-built instruction dataset called mR2AG-IT.
Multimodal retrieval-augmented generation exists because today's vision-language models carry frozen, limited knowledge and hallucinate on specifics like species names or landmark dates. But bolting a retriever onto a model tends to trigger needless searches, blur which evidence supports which claim, and pile on extra filtering modules. mR2AG's pitch is doing the query-gating and evidence-locating inside the model itself, with two lightweight reflection steps instead of separate machinery.
It is still a benchmark result on arXiv, not a shipped product, and INFOSEEK-style leaderboards have a habit of not translating cleanly to messy real-world photos and questions.