LLMs trained on clinical knowledge still can't do clinical reasoning when it counts.
Researchers tested whether feeding large language models a structured Cognitive Behavioral Therapy framework would change how those models actually respond to users in distress. It didn't, not really. The models scored as high as 96% on CBT licensing exam questions — demonstrating solid theoretical knowledge — but when deployed in dialogue, they collapsed into a single mode: validate the user, reflect their feelings back, repeat. The team introduced a metric called Protocol Leverage Force to measure how far a structured prompt can push a model away from its default behavior. Across three open-weight models and 14 clinical case studies, even the best intervention shifted behavior by roughly 1.2 to 1.3 percentage points.
The gap matters because the mental health AI space is crowded with products leaning on exactly this assumption — that an LLM briefed on therapy frameworks will behave like a therapist. This paper gives researchers a concrete instrument to test that claim and, so far, the claim fails. Knowing what Socratic Questioning is and choosing to use it in the right moment are two very different skills.
The findings echo a broader pattern in applied AI: benchmark performance and real-world behavior routinely diverge. A model that aces a licensing exam is a model optimized for the format of licensing exams, not for the messy, context-dependent work those licenses are supposed to certify.