Diffusion language models have a caching problem that existing tools can't solve — and a new paper proposes a fix.
Current KV caching techniques, the workhorse behind fast, high-throughput LLM serving, assume that once a token's key-value pairs are computed, they stay fixed. That assumption holds for autoregressive models like GPT-style transformers, but it breaks entirely for diffusion language models. DLMs use bidirectional attention, meaning every token can attend to every other token simultaneously. Update one token during the diffusion process and the entire context shifts — along with every cached value. The researchers found that naively applying standard prefix caching to DLMs causes model accuracy to collapse to near zero.
The paper introduces bicache, which works around this by exploiting a structural property the team identified: shared prefix key-value pairs stay stable and reusable in the shallow layers of a DLM, even as deeper layers churn. The safe depth isn't fixed — it scales with what fraction of a given request consists of shared prefix tokens. So bicache dynamically calculates how deep it can safely reuse cached values on a per-request basis, skipping redundant computation only where it's safe to do so. In evaluations, that translated to throughput gains of 36.3% to 98.3% over existing techniques, with accuracy differences of just 0 to 1.8%.
DLMs have attracted serious research interest as an alternative architecture to autoregressive generation, but production serving has remained impractical at scale. Throughput is the chokepoint that makes or breaks deployment economics, and bicache is the first technique to address it for this model class specifically.
Whether bicache gets adopted depends on whether DLMs ever graduate from research curiosity to production deployment — a question the architecture's boosters have been optimistic about for a while now.