Medical AI can pass every automated accuracy check and still give you the wrong drug's data.
Researchers tested 13 language models on a controlled benchmark and found that retrieval-augmented generation systems can produce responses with zero hallucinations, near-perfect faithfulness scores, and real citations — while quietly swapping the clinical evidence of one drug for another. They call this "deceptive grounding": the model fetches a real document, cites it correctly, but applies it to the wrong entity. Failure rates ranged from 8% to 87% under adversarial conditions. Notably, medical and biomedical fine-tuned models performed worse, not better, hitting deceptive grounding rates as high as 86.7%. In a deployed system evaluated across 740 drug-disease pairs, the overall rate was 7.8% — rising to 13.6% for recently approved drugs, where training data is thinner and retrieved evidence is more likely to be sparse or mixed.
This matters because the entire premise of RAG in clinical settings is that grounding responses in real documents makes them safer. If the documents are real but misattributed, that safety guarantee evaporates — and none of the standard evaluation frameworks catch it. The researchers found that a targeted fix, entity-attribution verification, detects the failure at 97% precision and 98.7% recall, but note that no existing framework actually implements it.
The finding lands at an awkward moment: health systems and drug-information platforms are actively deploying RAG-based tools as a credibility upgrade over raw language model outputs. Domain fine-tuning, often sold as the responsible path, appears to make this specific failure mode worse — which suggests the field has been optimizing for the wrong metrics.