Four open-weight language models are far more likely to refuse a harmful request in English than the identical request in Somali.
Researchers evaluated Llama-3.1-8B-Instruct, Gemma-2-9B-Instruct, Qwen-2.5-7B-Instruct, and Aya-23-8B against SomaliBench v0, a native-author-verified set of 100 harmful-intent prompts paired in English and Somali. Each model ran locally at temperature 0 under the same English "helpful, harmless, honest" system prompt. All four showed a refusal gap between the English and Somali versions ranging from 0.40 to 0.93, and the gaps held up under paired bootstrap and exact McNemar significance tests. A pinned Claude Sonnet snapshot classified each response as refused, complied, or unclear; it declined to classify 34 of 800 responses, which the paper's native Somali-speaking author labeled by hand. In a separate check, that same author reviewed 74 comparable rows and agreed with the classifier's judgments 100% of the time, a Cohen's kappa of 1.00.
For three of the four models, the Somali refusals didn't fail by producing fluent harmful answers. They failed by producing wrong-language, incoherent, or off-topic text, a different problem than compliance but one a simple refusal-rate count can hide. A safety system that looks intact in English can be effectively absent in a language its developers never tested, and nobody monitoring English-only evals would catch it.
Every one of these models ships with the same English-language safety promises attached. This study is a reminder that almost nobody checks whether those promises survive translation; the researchers aren't releasing the raw generations, since some Somali outputs may contain harmful content.