Arabic language models look safer on paper than they are once you switch dialects.
Researchers built SalamahBench, a benchmark of 8,270 human-verified harmful prompts, each translated into Modern Standard Arabic and five regional dialects (Egyptian, Syrian, Saudi, Lebanese, and Moroccan), yielding 49,620 paired test cases across ML Commons hazard categories. They evaluated models including Fanar 2, ALLaM 2, and Karnak 1 under multiple safety-guard configurations. To measure the dialect effect, the team introduced two new metrics: Dialect Shift, which tracks how much a model's overall safety changes when prompts move from MSA to a dialect, and Category-Specific Dialect Deviation, which flags harm categories that swing differently than the aggregate trend.
The results show dialect robustness doesn't track cleanly with a model's overall safety score. Some models hold steady in Modern Standard Arabic but get noticeably easier to jailbreak once prompts shift into dialects like Moroccan or Lebanese, and the slip isn't even across harm categories, which is exactly the kind of divergence the paper's new metrics were built to catch.
It's the same gap that trips up English-language red-teaming when testers move from formal text to slang, except in Arabic the everyday spoken dialects and the written standard most models are trained on can diverge even further, and this paper suggests most safety evaluations have been missing that gap entirely.