A new paper proposes a fix for one of speculative decoding's quiet cost problems: the helper model that's supposed to speed things up gets more expensive the longer your prompt gets.
The work, posted to arXiv and not yet peer reviewed, describes a system called LongSpark. Speculative decoding speeds up text generation by having a small drafter model guess several tokens ahead, which the full model then checks in a single pass instead of generating token by token. The problem: as conversations or documents grow, today's best drafters have to carry a growing memory of the entire prefix, so they slow down too - quietly eating into the speed gain they exist to provide. LongSpark's drafter instead pulls fixed-size snapshots from the main model's own verification step, so its cost stays flat no matter how long the input gets.
That matters because long-context work - RAG pipelines, coding assistants scanning whole codebases, hours-long chat sessions - is exactly where speculative decoding's edge has been quietly disappearing, by the paper's own account of the problem. A drafter whose cost doesn't scale with context is a real lever for anyone running inference at scale, not just a benchmark trick, if it holds up in production serving stacks outside the authors' own tests.
Speculative decoding only became a mainstream inference trick in the last couple of years, and most efficiency gains since then have come from tuning drafter architecture rather than questioning what a drafter needs to remember at all - which is the more interesting move here than the raw speed numbers.