A research paper out of arXiv proposes a scheduling fix for a latency problem that emerges when you split large language model inference across separate GPU pools.
Modern high-throughput LLM deployments often run "prefill" and "decode" on different GPU clusters to stop the two phases from stepping on each other. The problem: under bursty traffic, prefill nodes back up while decode nodes sit partially idle. On a test cluster with two prefill and two decode A100 nodes, the researchers found that actual prefill compute accounts for only 2-23% of the 95th-percentile time-to-first-token. The rest is queuing and the cost of moving key-value cache data between nodes over the network. Their fix is a proactive scheduler that redirects overflow prefill work onto decode nodes as small interleaved chunks, sized carefully so in-flight decode batches still meet their latency targets. Because the prefill runs locally on the decode node, the inter-node data transfer disappears entirely.
The results on production-style traces using DeepSeek-V2-Lite are notable: up to 81% reduction in P95 time-to-first-token and up to 79% improvement in service-level objective attainment, at under a millisecond of routing overhead per request. For inference providers billing by the token and competing on responsiveness, shaving tail latency without adding hardware is a meaningful lever.
The work builds on vLLM, the open-source inference engine that has become a de facto substrate for this kind of systems research — which means productionizing it is at least plausible, not just a paper exercise.