Researchers built a benchmark that catches AI health chatbots giving the right answer for the wrong reasons.
A team designed a four-part evaluation framework for testing how well large language models reason about medical interventions, not just whether they land on the correct one. The system pairs a cardiovascular causal knowledge graph, where every claim is tracked back to its source, with four test conditions that vary how much of that graph a model sees before answering. Across scenarios built to trigger eight distinct reasoning failures, the researchers scored each condition on intervention accuracy plus how well the model's stated causal links, side-effect claims, and evidence held up. The model given no graph context scored highest on raw accuracy, correctly identifying treatments 94.8% of the time, but showed essentially no grounding in causal mechanisms or supporting evidence. The model given the fully integrated graph scored lower on raw accuracy but scored best on every grounding measure: 83.8% causal-accuracy, 83.3% adverse-effect accuracy, 73.8% evidence accuracy, and just an 11.4% rate of unsupported claims.
That gap is the real finding. Most healthcare AI evaluations still reward a single correct answer, and this pilot shows that metric can reward confident guessing over actual clinical reasoning. In a decision-support tool, knowing why an intervention works, and what it might harm, matters as much as naming it correctly.
It is one pilot on one condition, cardiovascular disease, built by the same team proposing the benchmark, so treat the numbers as a proof of concept rather than a verdict on any deployed medical AI.