One language model, three audio cleanup jobs — that is the pitch behind UniSE.
Researchers published a framework that routes speech enhancement tasks through a single decoder-only language model rather than training separate systems for each job. UniSE handles speech restoration, target speaker extraction, and speech separation by generating discrete audio tokens autoregressively, conditioning the model on input speech features so it can switch between tasks without architectural changes. The team also layered in a progressive reinforcement learning strategy that scores outputs on multiple quality criteria to push audio fidelity higher. Code and demos are available via Alibaba's unified-audio repository on GitHub.
Speech enhancement has historically been a fragmented field — one model strips noise, another isolates a specific voice, a third untangles overlapping speakers. Collapsing those into a single LM-based system matters because it cuts deployment complexity and hints that the token-prediction machinery powering text models transfers more cleanly to audio than previously assumed. The benchmark results show competitive performance against both discriminative and generative baselines, which is a meaningful bar given how mature those approaches are.
Alibaba's audio research team is not alone in chasing audio-language model convergence — Meta, Google, and a handful of startups have each staked ground here — so "competitive" is doing a lot of work in that abstract. Whether UniSE holds up outside controlled benchmarks, in real call-center noise or live captioning pipelines, is the question the paper leaves open.