Vision-language models used to detect synthetic medical images can be tricked into reversing their judgments just by changing the text that accompanies an image - no pixel altered.
Researchers auditing multimodal robustness found that when VLMs receive both an image and a patient record or metadata, they often lean too hard on the text side of the equation. The same image, held fixed, received different authenticity verdicts depending solely on what the accompanying record said. In the starkest result, slapping an explicit AI-origin tag on the metadata caused accuracy on real, authentic images to drop by 61.1 percent on average across the models tested. The team evaluated a range of open-weight and commercial frontier models across multiple imaging modalities.
This matters because real clinical deployments do not feed images into AI systems in isolation - doctors review scans alongside structured records, notes, and metadata, and AI tools are increasingly built to do the same. A detection system that can be nudged toward a wrong call by a single metadata field is not a reliable guard against diagnostic fraud or insurance abuse. The researchers also propose an inference-time fix that flags and neutralizes these provenance shortcuts without requiring the model to be retrained, which outperforms simply prompting the model to ignore context.
The work arrives as health systems race to deploy AI-assisted imaging tools, and it is a useful reminder that a model passing a vision-only benchmark may still be a liability the moment a record gets attached.