Researchers have found a reliable way to make safety-trained AI models say things they're supposed to refuse — just ask in a language they weren't trained to refuse in.
A team studying multilingual robustness in large language models introduced STEER, a gradient-guided attack that pinpoints which words trigger a model's refusal behavior, then iteratively swaps those words into low-resource languages until the refusal collapses. Tested across six open-source 8-billion-parameter models, STEER achieved attack success rates of up to 93.0% on JailbreakBench and 96.7% on AdvBench — both standard red-teaming benchmarks. Crucially, prompts crafted against open-source models also transferred to GPT-4o-mini with a 35.5% success rate, no direct model access required.
The underlying problem is structural: safety alignment for most major models is conducted predominantly in English, leaving a gap the researchers call an "epistemic" one — models are confident they're behaving safely even when they aren't, because the harmful input simply falls outside what they learned to recognize as dangerous. That transferability finding is the sharper edge here: an attacker who fine-tunes an exploit on an open model they can freely probe may carry a meaningful success rate into closed commercial systems.
Safety evals have long focused on English-language adversarial prompts, and the jailbreak community already knew code-switching could weaken guardrails. What STEER adds is a systematic, gradient-guided method that outperforms both random code-switching and the well-known Greedy Coordinate Gradient approach — making this less a curiosity and more an automated toolkit. The fix the researchers propose, broader multilingual coverage during alignment and out-of-distribution abstention mechanisms, is straightforward to describe and considerably harder to ship at scale.