Researchers have built a vision-language model that deliberately throws away most of what it sees in a medical image — and performs better because of it.
The system, called ViToS, uses reinforcement learning to train a single model across two parallel tasks: one branch locates the clinically relevant region of an image, and the other answers medical questions using only the tokens from that region. The trick is a training method the authors call cross-feedback sequential optimization, which prevents the two objectives from fighting each other during learning. Tested across seven medical benchmarks, ViToS reduced the number of visual tokens processed to 77% of the original — a 23% cut — while scoring 108% relative performance on one baseline model and 104% on another.
This matters because medical images are not like everyday photos. An X-ray or histology slide may contain a tiny lesion buried in largely uninformative tissue; feeding a model the whole image is noisy and slow. Pruning irrelevant visual data before reasoning is a cleaner approach than hoping the model learns to ignore noise on its own, and the speed gains at inference time have real consequences for clinical deployment costs.
Vision-language models have advanced quickly in general domains, but medical imaging has lagged partly because labeled data is scarce and errors are costly. ViToS does not solve either of those problems — and performance numbers reported on academic benchmarks rarely survive contact with a hospital's actual data distribution — but the architecture at least attacks the right bottleneck.