Researchers say the standard approach to multimodal retrieval-augmented generation wastes money by reaching for a vision-language model too early.
The paper, posted to arXiv, targets a specific design flaw in how most multimodal RAG systems decide when to use images. Today's pipelines typically make that call upfront: either run a cheap text-and-table pipeline for everything, or pay for a vision-language model on every image retrieved. Newer adaptive systems try to route smarter by predicting before retrieval which modality a question will need. The researchers argue that prediction point is wrong. Using an oracle analysis on the MultiModalQA benchmark, they found that a question having an image in its gold-standard evidence set does not mean the image is actually required to answer correctly — many such questions can be answered from text and tables alone. Pre-retrieval routers that escalate based on apparent visual relevance end up calling the vision-language model far more than necessary.
The fix they propose — post-hoc selective modality escalation — flips the order: answer cheaply from text and tables first, then run a verifier on the draft answer and evidence to check whether something is missing, and only then pay for a vision-language model call if a calibrated router judges the accuracy gain worth the cost. On MultiModalQA, this approach matches the accuracy of always-on vision-language model pipelines while issuing significantly fewer visual calls. The practical implication is non-trivial: vision-language model inference is expensive at scale, and a reliable signal for when to skip it would meaningfully reduce operating costs for anyone running multimodal RAG in production.
The insight generalizes a known principle in RAG design — that routing decisions should happen as late as possible, with as much context as possible — to the modality dimension, sitting alongside earlier work on routing for retrieval depth and reasoning hops. Whether this holds on benchmarks beyond MultiModalQA is a question the field will need to answer.