A research team has built a way to shrink the image side of vision-language retrieval systems without throwing away the details that make searches work.
Most vision-language retrieval models keep a dense set of tokens — small chunks of image data — so that queries can match against fine-grained visual evidence. That thoroughness is expensive: storing and scoring all those tokens adds up fast. Existing compression methods trim the token count, but they tend to blur or discard object-level detail in the process. SaMer, proposed in a new arXiv paper, takes a different approach: it merges tokens into 64 representative clusters, using object annotations during training to prevent the model from accidentally blending features from different objects into a single token. At inference time, no bounding boxes or object detectors are needed. The result is a 93% reduction in image-side tokens and a 16.09x drop in storage for the ColPali retrieval system, while R@1 scores on the Flickr30K and MSCOCO benchmarks actually improve.
The finding matters because it reframes what compression should optimize for. It is not enough to reduce token count — the surviving tokens need to carry the right evidence for whatever query might arrive later. SaMer also shows stronger phrase-level grounding than competing baselines, which hints at broader usefulness beyond just storage savings.
Vision-language retrieval is quietly becoming infrastructure for document search, multimodal RAG pipelines, and enterprise knowledge bases — so a 16x storage cut that does not hurt accuracy is the kind of engineering result that tends to get adopted quickly, regardless of whether the paper ever makes a conference keynote.