A new benchmark called MedCalc-Pro aims to expose how badly AI models stumble when medical calculations get complicated.
Researchers published MedCalc-Pro to fill a gap in how AI is evaluated on clinical math. Existing benchmarks mostly hand a model one patient case, one calculator, and a clearly labeled question — nothing like the ambiguity of a real hospital workflow. MedCalc-Pro spans 2,268 real-world clinical cases, 77 medical calculators, and 14 clinical departments. It tests three levels of difficulty: single-calculator queries, multi-calculator problems that require joint evaluation, and nested calculations where one formula feeds into another.
The harder task settings matter because they reflect how clinicians actually work. A physician estimating drug dosage for a patient with kidney disease might need a creatinine clearance calculation nested inside a dosing formula — not two separate, labeled prompts. Benchmarks that skip that complexity give a false picture of AI readiness in clinical settings.
The same team also released an agent framework designed to handle multi-tool selection and nested tool calls, with structured validation to reduce the compounding errors that tend to snowball through chained calculations. It outperformed open-source, closed-source, and medical-specialized models across all three task settings. The caveat worth noting: the authors are also the framework's builders, so independent replication will matter before anyone treats this as a clinical endorsement.