A new shared task benchmarking Arabic-language content moderation systems shows they can spot obviously harmful chatbot prompts almost perfectly, but still fumble hateful memes.
ArGuard is a shared task pitting AI systems against two separate problems: spotting hateful content in Arabic-language memes, and flagging harmful prompts aimed at Arabic large language models. Fifty-eight teams signed up, 35 made it through to the final evaluation, and 27 wrote up their approaches in system-description papers. Entrants leaned on models including AraBERT, Jais, and Qwen3-VL. The task split into four sub-benchmarks, and the best-performing systems scored macro-F1 of 0.823 and 0.419 on the two meme tracks, and 0.984 and 0.790 on the two prompt tracks.
That gap is the real story. Catching an obviously harmful LLM prompt is nearly solved for Arabic - a 0.984 score is about as clean as these benchmarks get. But fine-grained meme classification landed at 0.419, dragged down by sparse labels and a mismatch between training and test data. Memes mix images, text, sarcasm, and cultural context, and Arabic dialects vary enough that a model trained on one region's slang can misread another's joke as a threat, or the reverse.
Multimodal hate-speech detection is still shaky even for well-resourced languages like English; Arabic, spoken across dozens of dialects with far less annotated training data, was always going to lag further behind. ArGuard's numbers just put a figure on how far.