Vision-language models tasked with rating medical image quality are easily fooled by pixelation and institutional name-dropping, according to new research.
Researchers tested 16 VLMs across seven imaging modalities using the MediMeta-C dataset, probing how each model held up against seven corruption types at five severity levels. Pixelation hurt the most: average scores dropped 20.58%, and OCT images saw reductions as steep as 34.4%. Brightness changes barely registered at -0.81%. More unsettling, the same corrupted mammography images pushed some models' scores up by 31%, suggesting the models were not just degrading gracefully but actively misfiring. Textual metadata compounded the problem: framing a case as coming from a high-prestige institution inflated scores by 17.15%, while describing older equipment dragged them down 14.7%.
The stakes here are not abstract. Medical Image Quality Assessment determines whether a scan is fit for clinical decision-making; a model that rewards institutional reputation instead of image clarity is encoding existing healthcare inequities into an automated gatekeeper. The pixelation finding is especially awkward because pixelation is a standard privacy-preserving technique — meaning the tools used to protect patients actively undermine the reliability of AI quality checks.
The researchers note that same-family models correlated at 0.67 to 0.83, which sounds reassuring until you consider the outliers: InternVL-8B swung +95.62% and MedGemma hit -37.7% on corrupted inputs. Deploying any of these models in a clinical pipeline without accounting for metadata bias and corruption sensitivity would not be a shortcut — it would be a liability.