A research team has open-sourced an inference pipeline that lets vLLM run unified speech understanding and generation without the usual throughput penalty.
Large multimodal models have gotten good at understanding audio, but the high-throughput engines used to serve them were built around single-stream text decoding. Speech generation is messier: it often requires multiple layers of audio tokens produced through a mix of autoregressive and non-autoregressive steps, with tokens interleaved in patterns that standard loops cannot handle. The new pipeline extends vLLM's autoregressive decoder to manage that interleaving natively, adds coordinated multi-stream sampling, and plugs in an on-GPU acoustic decoder so waveform synthesis happens end-to-end without leaving the GPU.
The more interesting claim is what they did with Classifier-Free Guidance. CFG is a standard technique for improving output quality, but it works by running two inference passes — one conditional, one unconditional — then merging the results. The conventional wisdom is that this halves your throughput. The team sidesteps the problem by co-scheduling both requests inside the same continuous batch, absorbing the overhead well enough to sustain roughly 80% of baseline throughput with CFG enabled.
That 80% figure matters because it shifts CFG from a quality-versus-speed tradeoff into something closer to a free lunch — at least compared to the naive implementation. Speech AI inference is still far more resource-hungry than text, and vLLM is already the de facto serving engine for open-weight LLMs, so lowering the barrier to deploying audio models on the same stack is a practical win. Whether the approach holds at production scale, with real traffic patterns, is a question the paper leaves open.