A new benchmark for healthcare AI agents sets a high bar — and current models clear less than half of it.
Researchers introduced HealthAgentBench, a suite of 54 tasks across seven clinical workflow categories, each with its own environment. The benchmark is designed to replicate end-to-end clinical work: agents receive minimal instructions, then must navigate raw healthcare data and execute multi-step solutions. Task categories span diverse modalities along the patient journey, and performance is reported as a single task success rate. The benchmark code is available on GitHub under Microsoft's account.
The results expose how far AI agents remain from practical clinical deployment. The top-performing agent achieved only around a 42% success rate — meaning it failed on more than half the tasks. Medical imaging proved especially difficult across models, while tasks combining large search spaces with compositional reasoning defeated every agent tested.
HealthAgentBench joins a growing pile of benchmarks designed to pressure-test AI in high-stakes domains. The difference here is the clinical specificity: rather than knowledge quizzes, these are workflow simulations — closer to what a deployed agent would actually face. A 42% ceiling on the best available model is a meaningful data point for anyone weighing near-term clinical AI deployment, and a useful corrective to vendor claims that frontier models are ready for healthcare.