A research benchmark called Indi-RomCoM exposes a blind spot shared by nearly every major language model: the fluid, mixed-language typing style used daily across South Asia.
Researchers built the benchmark to test how well LLMs handle Romanized Code Mixing - the habit of writing Hindi, Bengali, Tamil, or other Indic languages in Roman script while weaving in English words. The benchmark covers seven instruction-following tasks, four Indic languages, and three levels of code-mixing density. Proprietary, open-weight, and Indic-focused models were all tested under zero-shot and few-shot conditions. Every category of model underperformed, and performance dropped further as the proportion of code-mixing increased.
The finding matters because Romanized Code Mixing is not an edge case - it is the dominant way multilingual communities communicate in text across messaging apps, social media, and search. A model that falls apart on this input style is effectively less useful for hundreds of millions of everyday users, no matter how well it scores on English or native-script benchmarks. One partial bright spot: reasoning tasks degraded less than detection tasks like toxicity classification, apparently because generating an explanation forces the model to reconstruct context.
The gap between benchmark performance and real-world linguistic diversity is a recurring theme in NLP research - and Indi-RomCoM is a pointed reminder that "multilingual" on a model card rarely means the languages people actually mix.