Fine-tuning a speech model can make it worse at understanding actual humans talking.
Researchers introduced Vividh-ASR, a benchmark designed to stress-test automatic speech recognition systems on Hindi and Malayalam across four tiers of audio complexity: studio recordings, broadcast audio, spontaneous conversation, and synthetic noise. The study focused on Whisper, a widely used multilingual model from OpenAI, and found a recurring problem: fine-tuning it for these low-resource languages improved performance on clean, read speech while degrading results on spontaneous audio — the kind of messy, real-world speech most users actually produce. To fix this, the team developed a training approach called reverse multi-stage fine-tuning, or R-MFT, which applies large parameter updates early in training and sequences examples from hard to easy rather than the reverse. That combination recovered 12 absolute points of word error rate globally and pushed further gains on the spontaneous tier.
The efficiency angle is worth noting. Using R-MFT, a 244M-parameter Whisper model matched or beat conventionally fine-tuned versions nearly three times its size at 769M parameters. Analysis of how the model changes internally during training showed that the effective schedules concentrated adaptation in the decoder while leaving the encoder's acoustic structure largely intact — which may explain why the gains held across noise conditions.
Most ASR benchmarks still treat "low-resource language support" as a binary: does the model handle the language at all? Vividh-ASR pushes past that by stratifying difficulty, which is a more honest measure of real-world utility. The benchmark and models are publicly released, so the next question is whether other labs building Indic language tools will actually use it — or keep training on studio audio and calling it a day.