AI agents that browse images to answer questions are dragging around a lot of dead weight.
Researchers built MM-ContextFold, a training-free framework for multimodal agentic retrieval, where agents call external tools to look up both text and images while working through complex questions. After studying about 10,000 task trajectories, the team found that once an agent pulls information out of an image and writes it into the text context, keeping the raw image around stops helping. It just adds noise, and can even hurt accuracy. MM-ContextFold responds by keeping one persistent, text-only context for high-level planning and spinning up short-lived branch contexts whenever an image is actually needed. Once that branch finishes its subtask, the outcome gets folded back into the main context as a short text summary, and the image and branch trace get discarded. Across seven multimodal retrieval benchmarks and five different backbone models, the method lifted average accuracy by 6.3 percentage points over the standard ReAct baseline while cutting working context length by 27.5%.
Multimodal agents look impressive in demos, but their appetite for context is a real cost problem. Every image an agent keeps around eats into the token budget and the compute bill, and per this study, it can eat into accuracy too. Text-only context compression has been a live research thread for a while; treating images the same way, by extracting what matters and throwing away the pixels, is a more overdue fix than it sounds.
It's training-free and holds up across five backbones, which is a decent sign, but a 6.3-point bump on benchmark trajectories is not the same as a bump in a production agent juggling a messy real-world tool stack.