Serving a large language model efficiently is no longer just about how fast one GPU can crunch numbers.
A new systems paper synthesizes peer-reviewed research, open-source code, and production deployment reports to argue that LLM inference is moving from a single-engine optimization problem into a distributed control problem. It frames vLLM and llm-d as two layers of the same stack: vLLM's engine-level tricks, PagedAttention, continuous batching, quantization, and parallelism, squeeze more throughput out of a given GPU, while a control-plane layer decides where and when that execution should happen across a whole fleet. The authors propose an "Inference Execution Planner" that picks entire execution plans rather than just an endpoint to hit, weighing choices like aggregated versus disaggregated serving, how to source and move cached key-value state, which hardware variant to use, and how to route and admit requests. The paper also offers a benchmark atlas, a taxonomy of how bottlenecks migrate through a system, and an evaluation approach built around service-level-objective goodput rather than raw throughput. Notably, it introduces no new benchmarks of its own; every performance number traces back to its original source.
That reframing matters because it changes what "scaling inference" means for anyone running models in production. The implicit argument is that buying more accelerators solves less than it used to; the scarcer resource is now managed state, reusable KV cache, and network movement between nodes. It is the same lesson distributed databases learned decades ago: once a single machine's compute stops being the constraint, the real engineering work moves to the scheduler.
Still, this is a research synthesis and a proposed planning framework, not a shipped system with numbers to argue over. Whether an execution planner like this actually beats hand-tuned routing in production is the open question the paper itself defers to future work.