Cluttered images trip up vision-language models, and a new attention trick fixes that without retraining.
Researchers studied how vision-language models allocate attention across layers and found that visual complexity strongly correlates with scattered, high-entropy attention, which hurts reasoning accuracy. Attention starts broad in shallow layers and narrows in deeper ones, but the more cluttered the image, the less that narrowing converges. The team then compared attention maps generated for generic prompts against those for task-specific prompts, using the difference between them to separate genuine visual signal from background noise. That insight became CARVE, short for Contrastive Attention Refinement for Visual Enhancement, a training-free method that works directly on a model's existing attention weights.
No fine-tuning, no external segmentation tools, and no extra training data are required, just a sharper read of attention a model already computes. In testing, CARVE improved performance on open-source vision-language models by as much as 75 percent, a large gain for a fix applied entirely at inference time.
Most fixes for VLM visual reasoning add cost: more parameters, more training data, more inference-time tool calls. CARVE's bet is that the fix was sitting in the attention weights all along, though how it holds up beyond benchmark conditions remains to be seen.