A new inference framework called TIDE keeps retraining its own speed-up model while a large language model is running, instead of freezing it and hoping it still matches the traffic.
TIDE targets speculative decoding, the trick where a small 'draft' model guesses several tokens ahead and a bigger target model checks them in one pass. Draft models are normally trained once, offline, and go stale as the mix of prompts shifts, sometimes slowing inference down instead of speeding it up. TIDE fixes that by reusing the target model's own intermediate outputs, already generated during normal inference, as free training data for the draft model, so retraining adds no extra target-model computation. It also switches speculation on or off automatically when the payoff isn't there, and on clusters with mixed GPU types, it routes draft training to the cheaper cards while the target model keeps running on the rest.
Speculative decoding is already a common technique in LLM serving, but its weak point has always been drift: live traffic doesn't look like the offline data a draft model trained on. Treating that drift as a solvable operations problem, instead of leaving engineers to manually retrain and redeploy draft models, is aimed at people running inference at scale, not people chasing bigger models. The reported numbers back it up: up to 1.66x throughput over skipping speculation entirely, with the biggest gains on exactly the workloads where a static draft model used to make things worse.
The paper also claims 3.02x faster draft training and 24x less storage than existing methods, plus a 1.22x throughput bump on heterogeneous GPU clusters. Those comparisons are against research baselines rather than an existing production deployment, so the real test is whether any of this shows up in a serving engine near you.