Researchers have trained a speech language model that jointly handles meaning and sound without any text data or multi-stage token pipelines.
WavSLM starts from WavLM, a self-supervised model trained on raw audio, and distills those representations into a single codebook. From there, it trains an autoregressive model to predict the next chunk of that codebook in sequence. That sounds straightforward, but most competing systems are considerably messier: some depend on text transcripts to anchor semantic content; others run separate stages for semantic and acoustic tokens; still others layer multiple residual codebooks on top of each other. WavSLM collapses all of that into one stream.
The simplicity matters for practical reasons. Complex multi-stream architectures are harder to scale, more brittle to tune, and generate larger inference footprints. Text supervision is a crutch that ties a speech model's usefulness to the availability of transcripts - a real constraint for low-resource languages or noisy audio. If a single-stream approach can match established systems on standard benchmarks while using fewer parameters and less training data, it brings speech modeling closer to the scaling dynamics that made large text models so tractable. Streaming inference support is included, which is a prerequisite for any real-time application.
The paper is careful to say "competitive" rather than "state-of-the-art" - an honest hedge. Speech benchmarks are fragmented, and results on one set of metrics rarely transfer cleanly to another. The real test is whether this simplicity holds as the model scales up.