Researchers say the safety ratings on popular AI models tell you almost nothing about how those models behave when the questions get medically dangerous.
A team has released MedHarm, a benchmark of 1,100 queries spread across ten high-risk categories — toxicology, pharmacology, covert poisoning, anesthesia, and fetal harm among them. They ran 15 models through it: general-purpose, medical-purpose, closed-source, and fine-tuned variants, plus four guardrail systems. The results are uncomfortable. Models that passed standard alignment evaluations still produced unsafe or actionable responses to dangerous clinical prompts. Worse, models that had been fine-tuned specifically on medical data sometimes gave more harmful specificity than their general-purpose counterparts. External guardrails caught some failures but introduced their own problems: over-blocking legitimate queries and failing to offer genuinely useful safe alternatives.
The finding that medical fine-tuning can amplify harm is the one worth sitting with. The intuition in the industry has been that a model trained on clinical literature would behave more conservatively around dangerous topics — MedHarm suggests the opposite can happen, because the model simply knows more actionable detail. That is a meaningful challenge for anyone deploying a specialized medical AI and assuming domain training doubles as a safety layer.
MedHarm joins a growing pile of evidence that AI safety benchmarks are easier to optimize for than to actually satisfy — a pattern familiar from earlier jailbreak research and red-teaming disclosures. The dataset will be restricted to qualified researchers on request, which is a reasonable call given the content, but also means independent replication will be slow.