A new architecture paper argues that the two main tricks for speeding up language model generation can, in fact, be combined — but only if you constrain how far one of them looks.
The core problem is that generating long text is slow because each step forces a model to stream a growing pile of stored context from memory. Two camps have emerged with fixes: efficient attention variants and linear-time mixers like Mamba, which reduce memory access, and diffusion-style models, which generate whole blocks of tokens in parallel instead of one at a time. Earlier attempts to merge these approaches — like DiffuMamba — ran into a caching wall: Mamba's bidirectional scan reads the sequence in reverse, which means its internal state depends on tokens that haven't been decided yet, making it impossible to reuse cached results cleanly from block to block. The new proposal, called BDLM Mamba-H, fixes this by limiting the reverse scan to only the active block being denoised, not the whole sequence. That small scope change unlocks exact caching.
The results are notable for what they show at scale: at 65K tokens, BDLM Mamba-H reaches 19.7 times the throughput of full-sequence DiffuMamba-H; at 262K tokens, it still runs 3.7 times faster than a block-diffusion attention baseline. At 87M parameters, it posts the best validation perplexity in its sweep. The 350M-parameter version holds its own without dominating, which is a more honest result than most architecture papers volunteer.
Diffusion-based language models have lagged behind autoregressive ones in real-world adoption; the missing piece has arguably been inference efficiency at long contexts. This paper does not close that gap entirely, but it identifies a precise reason earlier hybrids failed and offers a targeted fix — which is more useful than another benchmark table full of green checkmarks.