Diffusion language models write several words at once to go fast, and a new paper shows that shortcut can break the math it depends on.
The paper looks at discrete diffusion samplers, including remasking and uniform-state variants, which fill in multiple token positions per step by drawing from a per-position guess and using those same guesses to decide which spots to fill first, usually the most confident ones. The authors prove that this only reproduces the real training distribution when the positions being filled are conditionally independent given what is already fixed. They also show no combination of per-position guesses can correctly capture cases where positions are dependent, and that the per-position numbers alone cannot even reveal whether that dependence exists. On a synthetic test called ScanAndAdd, where the true joint answer is known exactly, every multi-position write chosen by confidence ranking turned out to involve dependent positions, and the resulting output distribution was off by 29 times the sampling-noise floor.
That gap matters because speed is the entire pitch for diffusion language models over standard one-token-at-a-time transformers. This result says that pitch has a hidden cost: the parallel writes can be systematically wrong in ways that per-sample accuracy scores do not detect, since those scores in the test still read a perfect 1.0.
A model that looks flawless on every individual example while quietly getting the combinations wrong is exactly the kind of failure a leaderboard is built to miss.