A small change to how draft models are trained produces consistent speed gains in a technique used to accelerate large language model output.
Speculative decoding works by having a smaller "drafter" model propose a block of tokens that the main model then checks left to right, accepting only the longest valid prefix before stopping. The problem: most draft models are trained to get every token in a block right, even tokens that will never be used at inference because an earlier one failed. Spec-AUF, or Accept-Until-Fail, fixes this by restricting the training loss to only the tokens the drafter would actually get to keep — everything up to the first predicted failure. No auxiliary objectives, no extra rollouts, and no changes to how inference runs.
The gains are modest but consistent. On Qwen3-8B across six benchmarks, AUF raised the average number of tokens emitted per draft pass from 2.40 to 2.61 on the DFlash drafter architecture, with improvements on every benchmark tested. It also transferred to a different drafter architecture, Domino, moving that number from 2.56 to 2.68. Because speculative decoding is one of the main practical tools for reducing latency in production LLM serving, training improvements that require no inference-side changes are unusually easy to adopt.
The researchers also found that simply downweighting later positions in the block — a common existing technique — actually achieved higher raw token accuracy than AUF while producing worse decoding performance, a reminder that training metrics and real-world output quality can point in opposite directions. The result suggests the field has been optimizing the wrong signal for longer than it realized.