A new survey lays out, for the first time, a structured map of how the AI field actually converts messy multimodal data into the discrete tokens that language models process.
Researchers published a comprehensive survey of discrete tokenization for large language models, built around vector quantization (VQ), the technique that turns continuous multimodal data into discrete codes an LLM can ingest. The paper sorts eight representative VQ variants, spanning classical and modern approaches, and breaks down their algorithms, training dynamics, and the practical headaches of wiring them into LLM pipelines. It also surveys how these methods show up across three settings: classical applications without LLMs, LLM systems handling a single modality, and LLM systems handling multiple modalities at once. The authors flag persistent problems including codebook collapse, unstable gradient estimation, and modality-specific encoding constraints, and point to a public GitHub repository they plan to keep updating.
Tokenization is the unglamorous plumbing underneath every claim about multimodal AI capability, and this survey argues that plumbing is still leaky. If a quantization scheme collapses its codebook or produces unstable gradients, that failure quietly caps how well a downstream model can align, reason, or generate across modalities. A field-wide taxonomy matters because teams building multimodal systems have largely been solving the same tokenization problems in isolation, without a shared map of what has already been tried and what still breaks.
It is a survey, not a breakthrough, but it is the kind of housekeeping that tends to precede one.