Speculative decoding just got a tree upgrade for DeepSeek-V4, and it actually pays off in practice, not just on paper.
Researchers adapted tree-structured speculative decoding, a technique that keeps multiple candidate token sequences alive from a shared starting point, to the DeepSeek-V4-Flash pipeline. The catch: DeepSeek-V4's compressed attention system squeezes context into a shared state, which breaks when different branches need different states. The team fixed this with three additions - branch-aware verification, temporary state isolation, and a refresh step for accepted paths - so branches stop corrupting each other's compressed state. Tested across draft budgets of 5 to 8, batch sizes from 1 to 64, and three benchmarks (GSM8K, MBPP, ShareGPT), the tree method accepted more tokens per step than a matched linear baseline in every setting, and improved throughput in nearly all configurations, by up to 18.5 percent.
That 18.5 percent isn't evenly distributed. Gains grow with the speculation budget and matter most on unpredictable workloads at small-to-medium batch sizes, which is exactly where inference cost tends to add up fastest for anyone serving these models at scale. At the smallest budget, the speedup is close to a rounding error.
There's also a plateau: past a certain budget, throughput stops climbing even though the accepted token length keeps rising, meaning extra correct guesses stop translating into wall-clock savings. That's a useful check against reading "more accepted tokens" as "always faster." This is still an arXiv paper, not a shipped feature - the real test is whether inference providers bother wiring it into their serving stacks.