AI models tasked with summarizing clinical trial results make things up — and a new study tries to measure exactly how much.
Researchers built a benchmark framework using 200 clinical trials drawn from the ClinicalTrials.gov database, then generated summaries targeting three audiences: healthcare providers, patients, and payers. GPT-4o, Claude Sonnet 4.6, and Gemini 2.5 Flash each produced summaries scored against a six-dimension faithfulness schema using a cross-encoder natural language inference model. Across all 1,800 summaries, "Unsupported Claims" was the dominant failure mode, with a mean score of just 1.55 out of 3. The researchers then tested a knowledge-graph-augmented retrieval system against those baselines and found statistically significant improvements in faithfulness scores, with entailment rising by 0.0125 and overall faithfulness by 0.0130.
The study matters because clinical trial summaries are not low-stakes marketing copy — they inform treatment decisions, insurance coverage, and patient understanding. An AI that confidently fabricates a drug's efficacy rate is not a productivity tool; it is a liability.
What's notable is that the fix was not uniform: GPT-4o improved mainly by reducing contradictions, while Claude and Gemini improved by generating claims better supported by the source. That divergence suggests hallucination in medical AI is not one problem but several, and that a single mitigation approach will not cover the field.