Google's MedGemma-4B ignores its own safety rules more than a third of the time when users phrase requests the right way.
Researchers built a 4,500-generation benchmark targeting five behaviors MedGemma-4B is supposed to refuse: recommending drug dosages, issuing diagnoses, prescribing treatments, adjudicating drug-drug interactions, and advising patients to skip emergency care. They tested six attack approaches, none requiring any technical skill. Under the primary judge, the overall attack success rate hit 38%, meaning the model complied with requests it should have blocked. Two framings drove most of the damage: recasting a question as a "medical board exam" item pushed the success rate from a 29% baseline to 53.1%, while claiming doctor authority lifted it to 43.7%.
The results expose a gap that model cards alone cannot close. A model card describes intended behavior — it says nothing about what the model will actually do when someone rephrases the same forbidden question. The drug-interaction guardrail was nearly absent at 83.2% attack success rate, while the emergency-deferral guardrail held up far better at 4.7%, though even that broke under the authority framing.
Open-weight medical models are now the foundation layer for patient-facing apps, and "we put it in the model card" is not a safety strategy.