Researchers have a new way to measure whether AI can hold up its end of a conversation.
A team introduced SocialOmni, a benchmark designed to test omni-modal large language models — systems that process audio, video, and text together — on the social mechanics of real dialogue. The benchmark covers three specific skills: identifying who is speaking, knowing when to interject, and phrasing interruptions naturally. It includes 2,000 perception samples and 209 interaction-generation instances with tight temporal and contextual constraints. Twelve leading omni-modal models were tested.
The results expose a structural blind spot in how these models are built and evaluated. Models that scored well on perception tasks — correctly tracking speakers and following conversation flow — did not reliably produce appropriate interruptions when the moment came. That gap matters because most existing benchmarks reward understanding, not participation. A model that can transcribe a heated argument is not the same as one that can jump into one at the right moment.
Most AI evaluation still treats conversation as a comprehension test. SocialOmni is an early attempt to grade AI on the social contract of dialogue — a harder and more honest measure of how these systems will actually behave when deployed in real-time, voice-driven products. Whether the labs building those products will train against it is another question.