A new study finds prompting an LLM to judge doctor-family shared decision-making performs far worse than a simple supervised model - and worse still than human coders.
Researchers recorded 21 pediatric surgical consultations - 19 patients, about 6.1 hours of audio, 7,566 utterance segments - between families of children with multiple long-term conditions and their surgeons. Human coders tagged each segment for 12 shared decision-making behaviors, agreeing with each other at a Cohen's kappa of 0.695. The team then tested a zero-shot Qwen 2.5 32B model against a small supervised classifier trained on frozen sentence embeddings, plus a stacked combination of both, using patient-grouped evaluation folds meant to keep siblings' recordings and held-out patients from leaking into training. The zero-shot model managed a macro kappa of just 0.139; the supervised classifier hit 0.227; the stack reached 0.242.
That gap matters because zero-shot prompting is the default shortcut teams reach for when they want to score messy clinical conversations at scale - it's cheap and needs no labeled data. This paper shows that shortcut leaves a wide reliability gap versus even a lightweight supervised model, let alone the 0.695 kappa human coders achieved, in a domain where misjudging whether a family was genuinely included in a decision has real consequences for care-quality research.
The authors also caught their own pipeline leaking labels through few-shot exemplars, even after grouping by patient - a reminder that patient-level splits alone do not guarantee a clean evaluation, and that other published LLM-evaluation-in-healthcare results may deserve the same scrutiny.