A new evaluation tool for audio language models outperforms bigger, costlier judges on the task of grading open-ended answers.
Researchers introduced ORCA, short for Open-ended Response Correctness Assessment, a model-based scoring system built specifically for large audio language models. The team assembled 9,663 human annotations across 3,699 question-answer pairs, drawing from 15 different models tested on three audio benchmarks. They used a three-stage pipeline that blended human judgment, structured feedback, and human-AI correction to train ORCA. The result is a Spearman correlation of 0.91 with average human ratings on benchmarks it was trained on, and 0.85 on benchmarks it had never seen.
That generalization number is the part worth sitting with. Evaluation tools that fall apart on new data are a recurring problem in AI benchmarking, and a score of 0.85 on unseen benchmarks — while outperforming Gemini 2.5 Flash as a judge — suggests ORCA is capturing something more durable than surface pattern-matching. ORCA also models human disagreement, flagging benchmark items where raters diverged, which is a practical tool for identifying questions that are too ambiguous to be useful.
Audio AI benchmarks have historically lagged behind text equivalents in rigor, partly because open-ended audio tasks are harder to score automatically. ORCA does not solve that problem entirely, but it narrows the gap between human graders and automated judges — at a fraction of the compute cost of routing everything through a frontier model.