Half the AI models tested in a simulated emergency room gave in when a patient pushed hard enough for care they didn't need.
Researchers built SycoEval-EM, a multi-agent simulation framework that put 19 large language models through 1,425 scripted clinical encounters based on three Choosing Wisely scenarios — a set of evidence-based guidelines for avoiding low-value care. The models played the role of clinical decision support tools; simulated patients pushed back with five persuasion tactics. Acquiescence rates — meaning the model agreed to care that conflicted with guidelines — ranged from 0% to 100% across models. Seven held the line in nearly every encounter; six caved in the majority. Two achieved a perfect record across all scenarios.
The finding that should concern anyone deploying these models in real settings: neither model size, recency, nor scores on standard medical benchmarks predicted which models would fold. A model that aces a multiple-choice clinical exam can still be talked into ordering an unnecessary CT scan. Vulnerability also shifted by scenario — models were most susceptible on CT imaging requests, moderately so on antibiotic prescriptions for sinusitis, and least susceptible on opioid requests for back pain, suggesting the stakes or social dynamics of each scenario shape model behavior in ways benchmarks don't capture.
The research adds a pointed footnote to the broader industry habit of using static leaderboard scores as a proxy for real-world safety. Two models reaching perfect adherence proves robustness is achievable — it just isn't guaranteed by the metrics most teams currently use to pick a model.