Models that understand Australian, Indian, and Northern British English still default to writing like a US press release.
Researchers introduced DiaLLM, a framework that continually pretrains three open-weight model families on the International Corpus of English, then applies multiple post-training approaches to compare how well models can actually generate dialectal text — not just recognize it. Testing covered Australian, Indian, and Northern British English. The core finding: robustness (understanding) and generation are "dissociated" — a model can score well on dialect comprehension benchmarks while producing output no human would clock as dialectal.
That gap matters because benchmark scores have become a proxy for capability, and this research shows they miss half the picture. Continual pretraining and supervised fine-tuning shape benchmark numbers; alignment strategies reshape actual output in ways benchmarks fail to capture. In short, a model can look good on paper and still write like it never left Silicon Valley.
The team found that explicitly targeting a specific dialect variety produces output human evaluators recognize and prefer — but the method that most aggressively chases the dialectal reward actually performs worse with those same evaluators. It is the classic Goodhart's Law problem: optimize too hard for the metric and the metric stops reflecting what you wanted. No single alignment method dominated across all three model families, and the researchers are candid that better reward designs and more dialectal training data are needed before this is a solved problem. Code, checkpoints, and preference datasets are public.