A research team has released SCRIBE, a diagnostic framework that replaces the blunt word error rate metric with category-level error analysis for Indic automatic speech recognition.
Word error rate, the standard yardstick for ASR quality, treats every mistake the same — a misrecognized medical term counts the same as a dropped comma. SCRIBE splits errors into four buckets: lexical, punctuation, numeral, and domain-entity. It also addresses a structural problem specific to agglutinative languages like Malayalam and Kannada, where grammatically valid sandhi merges — word fusions that follow language rules — inflate WER scores unfairly, making a correct transcription look worse than it is. Human evaluators confirmed SCRIBE's rankings matched expert judgment in cases where WER diverged.
The gap matters because correction cost, not raw error count, is what determines whether ASR is actually usable. A system that mangles domain terms in a medical or legal context is far more expensive to fix than one that misses punctuation, yet WER lumps them together. SCRIBE gives developers the granularity to tell the difference and tune accordingly.
The team is releasing the full framework, an LLM curation pipeline, benchmarks, and open-weight rich transcription models for Hindi, Malayalam, and Kannada — a meaningful contribution given how underserved these languages remain in mainstream ASR research, where English benchmarks have driven the field for decades.