A new inference trick lets byte-level language models predict several bytes at once, without slowing training or growing the model.
Researchers built on multi-token prediction (MTP), a technique already used to speed up standard subword-based large language models, and adapted it for hierarchical byte-level models that read and generate raw bytes instead of tokenized text. The new method, called multi-byte prediction (MBP), uses a variable-length prediction window that lines up with a model's internal latent segments, plus an attention-masking scheme that lets bytes be predicted in parallel without breaking causal order. It adds no extra parameters. Tested on instruction following, question answering, summarization, and machine translation, the authors report it achieves the best available trade-off between output quality and inference throughput among the methods compared.
Byte-level models are pitched as a tokenizer-free alternative to standard subword models, useful for languages, code, or text that tokenizers mangle, but generating output one byte at a time has always been their speed tax. MBP is an attempt to close that gap without the usual cost of added parameters or degraded output quality.
It is a paper's claim, not a shipped product, and Pareto-optimal is doing a lot of work here. The real test is whether other labs can reproduce those throughput gains outside this paper's own benchmark set.