Syllabic speech tokenizers have a contamination problem: they were quietly learning speaker identity instead of linguistic content.
Researchers found that training syllabic tokenizers with an utterance-level cross-entropy objective causes the model to latch onto speaker characteristics rather than the underlying sounds and syllables it is supposed to capture. Their fix is a speaker-disentangled tokenizer that perturbs speaker information in the student representations and then regresses those representations toward clean teacher targets, all within fixed-length chunks. The method builds on the teacher-student distillation framework popularized by HuBERT, a widely used pretrained speech model, but adds a deliberate decoupling step to keep speaker identity out of the learned tokens. Tested on syllable boundary detection and syllabic segment clustering, the approach claims state-of-the-art results on both tasks.
The practical payoff shows up downstream: a speech language model trained on these cleaner tokens scores 7% better on syntactic and semantic understanding compared to phone-level SpiRit-LM, a prior benchmark. That gap matters because syllabic tokenization is pitched as a path toward speech language models that reason closer to how humans process language — in syllable-sized chunks rather than raw phonemes — and contaminated tokens quietly undermine that whole premise.
The result is a reminder that self-supervised speech models can optimize for the wrong signal without anyone noticing. Speaker identity is a strong, easy-to-learn cue; linguistic content is harder. Until someone runs the ablation, the model just learns what it can.