A new paper offers a shortcut for picking the right vision-language model without running it on your data first.
Researchers built a framework that scores how well a pretrained VLM will transfer to a new task by looking at the model's internal wiring rather than test-set accuracy. It measures layer-wise conductance - essentially how much each block of the visual encoder contributes to a task - then builds a profile of which blocks matter most for a target task. A new metric called Directional Conductance Divergence compares that profile against other tasks the model has already handled, in one direction only, since a model that's good at task A won't necessarily be good at task B the same way it's good at task C. Tested across 48 VLMs and 21 datasets, the approach outperformed multiple state-of-the-art selection baselines, including a 14.7% NDCG@5 improvement over one called SWAB.
That's a meaningful fix for a real bottleneck. Teams building products on open VLMs currently either run expensive few-shot evaluations across candidate models or lean on generic text descriptions of a task that ignore how transfer actually works inside the network. A method that predicts rankings without extra inference could save real compute for anyone shipping VLM-backed features.
Directional, asymmetric transferability makes intuitive sense - not every skill transfers both ways - but the real test is whether this holds up outside the benchmark suite. Beating multiple state-of-the-art baselines, including that 14.7% NDCG@5 edge over SWAB, is a solid showing on paper. Whether it survives contact with messier, real-world task definitions is a separate question.