A new benchmark says today's best AI models are still bad at doing the unglamorous math doctors do every day.
Researchers built MedMCP-Calc, a set of 118 test scenarios across four clinical domains, to check how large language models handle medical calculators the way clinicians actually use them: with vague requests, patient records to dig through, and multiple steps to get to an answer. Instead of just feeding a model clean numbers and asking it to compute a score, the benchmark forces it to pull data from a structured EHR-style database, look up external references, and choose the right calculator on its own. Testing 23 leading models, the researchers found that even Claude Opus 4.5, the strongest performer, struggled to pick appropriate calculators for open-ended queries, made errors during iterative SQL-based database lookups, and often avoided using external tools for calculations it could not do reliably itself. The team also built CalcMate, a fine-tuned open-source model aimed at this specific workflow, which topped other open-source models on the benchmark.
This matters because medical calculators are not a toy problem. They inform real decisions, from dosing to risk scoring, and the gap between demo performance and workflow performance is exactly where AI tools tend to fail quietly in production. A model that aces a textbook-style calculation prompt but can't reliably fetch the right patient value from a database first is not ready to be trusted unsupervised.
Benchmarks like this are becoming the pattern for AI-in-medicine research: less "can it answer a quiz question" and more "can it survive contact with messy, real-world data." That the field's top model still stumbles here is a useful reality check against the pace of announcements claiming AI is ready for clinical deployment.