Researchers have built a system that produces speech and synchronized facial movement simultaneously, instead of treating them as separate problems.
Existing conversational avatar systems split the work: one class of models handles real-time speech generation but ignores facial motion entirely, while another animates a face from audio that already exists. FacePlex, described in a new preprint, collapses both into a single streaming pipeline. At each step, the system produces speech tokens and facial motion tokens together. Two mechanisms do the heavy lifting: Rolling Flow Matching, which commits new motion frames continuously during streaming, and Rolling Cross-Attention, which lets the audio queue and motion queue inform each other as generation proceeds.
The gap FacePlex targets is real. Conversational avatars that can only lip-sync to pre-generated audio will always lag — the face responds to the voice instead of moving with it. Genuine joint generation is a prerequisite for avatars that feel like actual conversation partners rather than dubbed video. The researchers back their claims with ablation studies and a user study showing better lip-sync quality and motion fidelity than audio-driven baselines.
Avatar research has accelerated sharply as video-call fatigue and virtual-presence tools push demand, but most commercial products still bolt animation onto finished audio. FacePlex is a preprint, not a product — and turning a streaming research framework into something robust enough for real calls is a long road from an arXiv submission.