A single new neural network layer is enough to teach a text-to-video generator someone else's voice.
Researchers modified an existing text-to-audio-video model by bolting a zero-initialized linear layer onto its audio backbone, then fine-tuning it for a short training run. At inference time, the model takes a short reference recording and uses it two ways: the recording's diffusion latents get prepended to the audio stream, and a separate speaker embedding modulates the generated audio tokens throughout. Tested against five established voice-cloning text-to-speech systems on a set of 674 speaker-text pairs covering 30 speakers, the modified 5B-parameter model scored highest on speaker similarity across three separate verification tools. The paper also notes a practical bonus: the audio half of the model can run on its own, skipping video generation entirely for a roughly 30x speed boost when only voice output is needed.
The interesting part isn't the voice cloning itself - that's a crowded field - it's how cheaply it was bolted onto a model that wasn't built for it. One extra layer and a short fine-tuning pass turned a generic text-to-video system into a specialized voice cloner, which suggests other single-purpose capabilities might be similarly cheap to graft onto general-purpose generative models rather than trained from scratch.
Worth noting: the benchmark tops out at 30 speakers, and the paper doesn't say how the approach holds up with accents, languages, or voices well outside that sample. A cheap trick that works on 30 speakers is not automatically a cheap trick that works on anyone.