Getting a search engine to match a sentence to the right image — or vice versa — turns out to hinge on a surprisingly mundane math choice.
Researchers published a systematic study of how well different similarity metrics and neural architectures can align the embedding spaces produced by vision-language models and separate unimodal encoders. They tested four standard metrics alongside two learned ones, including a custom contrastive loss function, across multiple benchmarks. Cosine similarity — the workhorse of most embedding search stacks — came out on top across the board. Wasserstein distance, borrowed from optimal transport theory, didn't win head-to-head but offered a distinct view of how image and text distributions differ globally, which could matter for debugging retrieval pipelines.
The practical upshot: teams building cross-modal search or retrieval-augmented generation systems now have empirical backing for a default choice, rather than guessing. The finding that a custom contrastive loss outperforms mean squared error for both transformer and MLP architectures also gives practitioners a concrete architecture recommendation, not just a theoretical one.
Cross-modal retrieval sits at the foundation of products ranging from image search to video captioning to multimodal RAG pipelines — so "what metric should we use" is less an academic question than an engineering one that gets answered, badly, every day. The code is open-source, which at least means teams can test these conclusions against their own data rather than taking the paper's word for it.