A new paper argues that an AI model can replace human raters when scoring voice AI conversations, and might do it a hundred times cheaper.
Researchers tested Gemini 2.5 Flash as an automated judge for full-duplex voice agent conversations, rating recorded stereo audio directly on eight production-quality dimensions. They validated it against three calibrated human raters across 209 sessions: 152 conversations spanning 13 accent-and-condition variations and 57 clips with deliberately injected defects. On five of eight dimensions, the AI's scores tracked human scores within a margin of 0.07 Spearman rank correlation. On six of eight dimensions, it agreed with the human mean within one point on 60 to 92 percent of sessions.
Human evaluation at scale is expensive. The paper estimates AI-based judging runs roughly two orders of magnitude cheaper than staffing a human rating operation at the same cadence. That cost gap gives teams building voice AI a strong financial reason to automate, even where the method is imperfect.
The researchers also tested two additional Gemini models, with mixed results: one improved simple agreement to eight of eight dimensions, while another rated several dimensions markedly lower than humans despite similar rank-ordering ability. The takeaway is that a model swap still needs fresh calibration runs. "Probably fine" is not a methodology.