A new benchmark exposes how poorly AI agents handle real Southeast Asian language contexts beyond surface-level translation.
Researchers introduced SEATauBench, adapting the existing TauBench evaluation framework to five languages: Mandarin, Vietnamese, Thai, Indonesian, and Filipino. The benchmark tests agents across progressively localized settings — changing not just the conversation language but also tool specifications and task domains. Tested across three recent models, the results show that English agent capabilities hold up reasonably well when only the dialogue language shifts. But performance degrades sharply as more of the task context gets localized, with the steepest drops occurring under full domain adaptation.
The findings matter because "sovereign AI" — the idea that nations should have AI systems that work natively in their own languages and contexts — is a stated priority across Southeast Asia. Most agent benchmarks are English-first, which this work shows is a poor proxy for multilingual capability. A model that looks competent in English may quietly fail when the entire task environment is localized.
The paper also ships a reusable adaptation pipeline, meaning other researchers can extend the framework to additional low-resource languages without rebuilding from scratch. That's the more durable contribution here — a single benchmark result ages fast, but a pipeline that lowers the cost of building similar evaluations does not.