A research team has built a virtual psychiatric ward to find out how badly AI fails at real clinical work — and the answer is: considerably.
The benchmark, called MentalHospital, runs AI models through complete psychiatric encounters rather than isolated tasks like writing a diagnosis or drafting a treatment note. It draws on 1,193 de-identified patient records covering 76 disorders across all major ICD-11 categories, and structures each encounter around the standard clinical S.O.A.P. workflow — Subjective Interviewing, Objective Examination, Diagnostic Assessment, and Treatment Planning. To score results without bottlenecking on scarce human reviewers, the researchers also built MentalEval, a set of five AI evaluators trained to assess communication empathy, professionalism, note quality, diagnostic rigor, and treatment appropriateness. Those evaluators achieved a 0.944 weighted kappa score against expert raters — strong agreement.
The gap between AI and clinicians is the headline number: the best-performing model trailed human doctors by 37.28 percentage points on objective psychiatric competence. Mental status assessment — reading a patient's current psychological state in real time — emerged as the single biggest bottleneck. That matters because it is precisely the skill that resists rote pattern-matching; it requires reading affect, coherence, and context together.
Most AI psychiatric benchmarks test piece parts: can the model name the right diagnosis, or generate a plausible note? MentalHospital tests the full encounter, which is harder and more honest. Twenty-two clinicians rated the environment's fidelity at 3.88 out of 5 — decent, not glowing. Whether that ceiling on simulated fidelity limits the benchmark's real-world conclusions is a question the paper does not fully resolve.