Researchers have released Zarya, a language model that can generate text two different ways from the same trained weights.
The model, built in 0.6B, 1.7B, and 4B parameter sizes, blends two competing approaches to text generation: autoregressive models that write one token at a time in order, and masked diffusion models that can fill in multiple tokens in parallel. Zarya trains on both objectives at once, using a curriculum that starts with fine-grained autoregressive learning and gradually shifts to coarser diffusion-style learning across variable-size chunks of text. At inference time, the same trained model can run in either mode: standard diffusion-style parallel decoding, or a hybrid mode that uses diffusion to pick which chunk to fill next while generating the words inside that chunk one at a time, autoregressive-style. That hybrid mode is what lets it reuse the key-value cache, the memory shortcut that makes autoregressive models fast, which pure diffusion models normally cannot do.
Diffusion language models have been pitched as a way to break the sequential bottleneck of autoregressive generation, but they have struggled with both speed, since they cannot reuse cached computations, and coherence, since predicting many tokens at once means guessing at dependencies between them. Zarya's slot-based approach is an attempt to get parallel speed without giving up cache reuse or the step-by-step coherence that makes autoregressive models reliable. It's also notable that the training and inference setups are fully decoupled and reconfigurable: a model trained one way can be deployed in either mode, which matters more for other researchers building on this work than for end users.
Whether this hybrid approach beats a well-tuned autoregressive model on real tasks, rather than just avoiding the worst weaknesses of pure diffusion, is the open question the benchmark numbers alone will not settle.