Cutting corners on LLM inference accuracy turns out to cost more than researchers expected.
Speculative decoding is a technique that speeds up large language model output by running a smaller, faster model to draft tokens in advance, then having the main model verify them in parallel. The standard version is lossless — it preserves the original model's output distribution exactly. A newer family of approaches relaxes that guarantee, trading some fidelity for additional speed or capability gains without retraining anything. Researchers at arXiv surveyed these training-free relaxed methods, unified them into a common framework, and ran benchmarks against current models to see what actually holds up.
The findings complicate the pitch. Relaxed approaches introduce quality drift that standard speculative decoding avoids entirely — which means practitioners need to run substantial capability evaluations to know what they're giving up. That's a non-trivial overhead that marketing for these techniques tends to skip. Worse, many relaxed methods assume the draft model is itself a strong general language model, which rules out the lightweight dedicated multi-token-prediction drafters that are often the whole point of going cheap on the draft step.
Speculative decoding has become a quiet backbone of inference optimization across labs, with companies like Anthropic and Google describing variants in their own systems. This paper is a useful check on the "relax the rules and go faster" instinct — the speed gains are real, but so is the new evaluation debt that comes with them.