Researchers built a benchmark that finally separates a voice agent's brain from its ears and mouth.
Most voice agents are cascaded systems: a speech-recognition model transcribes what the caller says, a language model decides what to say and which backend tools to call, and a text-to-speech model reads the reply aloud. Existing tests either score the whole pipeline, mixing recognition mistakes with model mistakes, or test the language model in isolation, missing the messiness of real calls, split-up sentences, transcription noise, and specific language or script requirements. MTVA-Bench fixes that by putting the language model through the same conditions it faces on a live call: an LLM plays the caller according to a rubric, and a mock backend answers whatever tool calls the model actually sends. The benchmark spans 49 agents across 490 reviewed scenarios in seven languages, scored with deterministic checks on tool calls plus two LLM judges, one grading task-specific rules and one grading overall conversation quality, weighted equally.
In a seven-model study, six models picked the right tool within 6.4 points of each other, but their overall scores spread across 24.4 points. Tool selection, in other words, is basically solved. What separates a good voice agent from a bad one is how it fills in arguments, sequences actions, and talks while doing it.
That is a useful correction to demos that show a bot correctly booking a flight and call it done. Any call-center vendor claiming their bot "understands" callers should be tested on whether it still follows the rules mid-sentence, not just whether it dialed the right function.