Researchers have built a pipeline that manufactures fake conversations to teach AI models the hardest part of talking: knowing when to jump in and when to shut up.
A new paper describes a system that uses a large language model to script conversational events - who's speaking, what they say, whether it's an interruption, a brief acknowledgment, or an aside to a third party - without dictating exact timing. Those scripted events are synthesized into two-channel audio and placed on a shared clock, producing English and Mandarin speech covering 42 distinct conversational phenomena across eight categories. The team fine-tuned Moshi, an existing full-duplex speech model, on the resulting corpus. Afterward, Moshi correctly took its turn 85% of the time, up from 44% before, and its frame-by-frame accuracy at predicting who holds the conversational floor jumped from 0.46 to 0.88.
Full-duplex conversation - listening and speaking at once, the way humans do - has been voice AI's stubborn unsolved problem, and most assistants still either talk over you or leave dead air after a mid-thought pause. The gains here come from training and testing on data generated by the same synthetic pipeline, so how well the skill transfers to the chaos of real phone calls and meetings is still unproven. But labeled examples of pauses, interruptions, and backchannels are scarce in real recordings, and manufacturing them on demand sidesteps that scarcity directly.
Voice assistants that can take a hint without waiting for dead silence are now a data problem, not just a modeling one.