Researchers built a test to see whether an AI model actually knows when it's wrong about a medical diagnosis, and it mostly does, until the case gets murky.
Researchers created 45 synthetic patient vignettes comparing probable Alzheimer's-type dementia against depression-related cognitive impairment, then ran each one through three different prompt versions for 135 total trials. They tested GPT-4.1-nano, a small OpenAI model, asking it to pick a diagnosis and report how confident it was. The model got the diagnosis right 93.5% of the time and reported an average confidence of 78.4%. A calibration score called type-2 AUROC, which measures whether confidence actually predicts correctness, came out to 0.876, meaning the model's stated confidence lined up reasonably well with whether it was actually right.
This isn't just an accuracy number. It matters because a clinician using this tool would need to know when to trust the AI's confidence and when to double-check it. Confidence rose when evidence clearly pointed one way and fell when information was missing, which is the behavior you'd want from a well-calibrated system. But errors clustered specifically in ambiguous, conflicting Alzheimer's-leaning cases, where the model wrongly leaned toward depression and stayed more confident than the accuracy justified.
That's the failure mode that matters most in medicine: not being wrong, but being wrong and sure of it. A single calibration number can hide exactly the blind spot a clinician most needs flagged.