AI chatbots will hand a doctor a benzodiazepine taper schedule and refuse the patient who asks the same question.
A new benchmark called IatroBench tested six language models on 60 pre-registered clinical scenarios, each written to check for two kinds of harm: bad advice (commission) and withheld information (omission). Every scenario was run twice, once as a patient asking directly and once as a doctor consulting on the case. All five models scorable in this test shared more information under the doctor framing than the patient framing, a pattern the researchers call "framing-contingent withholding," with an average gap of +0.38 under their physician-written rubric (p = 0.003) and +0.22 under a separate LLM judge. Claude Opus 4.6 served as the paper's scoring model and matched a physician's judgments about as closely as a second physician would; GPT-5.2 had to be excluded from parts of the analysis because it returned no text for 33.2% of doctor-framed prompts while answering every single patient-framed one.
The result matters because it undercuts the excuse that heavy refusals are just caution. A standard LLM judge scored 86.6% of the cases the researchers flagged as omission harms as having zero harm at all, which means typical safety evaluations are largely blind to this failure mode. It also dents the pitch that chatbots level the playing field on medical knowledge: if a model already knows the answer and only shares it with the person holding a medical license, the software is replicating gatekeeping, not removing it.
Llama 4 complicates the story by performing poorly in both framings, so its smaller doctor-patient gap looks less like sensible caution and more like it doesn't know the answer either way, a useful reminder that safe and capable are not the same axis, whatever the model card claims.