Researchers have built a benchmark to measure something text evals consistently miss: whether AI voice systems actually converse like humans do.
SPEARBench evaluates streaming speech-to-speech language models — systems that take spoken questions and answer back in synthetic speech — using a multi-dimensional scoring protocol. It draws prompts from the Seamless Interaction corpus, runs them through several contemporary models, and scores the outputs across response latency, interruption behavior, speech quality, dialect consistency, emotional naturalness, and what the authors call interpersonal stance dynamics. Human answers are included as a reference condition, giving the benchmark a concrete ceiling to measure against.
The findings land a quiet rebuke at the current state of voice AI: models can score well on signal-level quality and transcription accuracy while still behaving nothing like a person in an actual conversation. Timing is off, dialect gets flattened, emotional adaptation lags, and the back-and-forth cadence that makes conversation feel natural is largely absent. Those are exactly the dimensions that existing text and speech benchmarks do not measure — which is why labs have been able to claim progress while the awkwardness in real use persisted.
The race to ship voice interfaces — from phone assistants to real-time translation tools — has outpaced the tooling for evaluating them. SPEARBench is an attempt to close that gap, though a benchmark is only as useful as the training signal it generates; whether labs treat naturalness scores as a target worth optimizing remains to be seen.