Most AI math benchmarks are written for speakers of two languages — and it shows.
Researchers have released PluraMath, an extension of the existing PolyMath benchmark that adds 18 underrepresented languages spanning six language families, from mid-resource to extreme low-resource settings. The dataset was built using a human-curated pipeline in which native speakers validated pre-computed translations rather than trusting automated output alone. The team then used PluraMath to benchmark 27 reasoning models across four size tiers — small, mid-size, large, and closed-source ensembles — to see how they hold up under genuinely diverse linguistic conditions.
The findings confirm what critics of English-centric AI development have argued for years: performance drops sharply for underrepresented languages, and better instruction-following ability — not raw reasoning power — is the strongest predictor of who closes that gap. That matters because it shifts the blame partly onto fine-tuning choices, not just training data volume, which means the problem is more tractable than it looks.
PolyMath, released in 2025, was already considered a step forward by covering 18 languages — all high-resource. PluraMath doubles that count by going where the existing benchmarks explicitly did not. The dataset, pipeline, and evaluation framework are fully open-sourced, which is the right call: proprietary benchmarks have a poor track record of driving broad progress on underserved communities.