A new benchmark says AI's diagnostic reasoning still lags far behind the marketing.
Researchers built DiagnosisArena, a test set of 1,113 segmented patient cases and their diagnoses, pulled from clinical case reports in 10 top-tier medical journals and spanning 28 medical specialties. The team screened the data with both AI systems and human experts to prevent models from simply recalling case reports they'd seen during training. They then ran current reasoning models against it. OpenAI's o3 scored 51.12% accuracy, o1 scored 31.09%, and DeepSeek-R1 scored 17.79%.
These are the same models getting pitched for clinical decision support, and the best of them barely clears a coin flip on real diagnostic reasoning. That gap is the difference between a promising research tool and something you'd actually trust in an exam room. The spread between models is also wide enough to suggest a genuine capability gap, not just noise in the test.
Remember this the next time a startup demo shows an AI nailing a rare disease on the first try - DiagnosisArena suggests that demo was probably cherry-picked.