Routing AI inference requests is unglamorous work — and most systems do it badly.
Researchers have proposed a multi-objective optimization framework that treats request routing in large language model serving as an online linear programming problem. When requests arrive, they get dispatched to parallel decode workers based on a calculated bid-price: if a request's service-level objective-weighted benefit clears a shadow price threshold, it gets admitted. The system updates those shadow prices continuously using a warm-started, projected first-order method fast enough to meet millisecond decision windows. Tests run inside the Vidur simulator showed improvements over standard baselines across end-to-end latency, time-to-first-token, throughput, and tail performance.
Most production LLM serving stacks — vLLM, SGLang, and similar frameworks — lean on routing heuristics that were never designed around explicit latency or throughput targets. That gap matters at scale: when KV-cache slots and batch sizes are the bottleneck, an ad-hoc round-robin or least-load rule leaves performance on the table and makes service-level guarantees hard to enforce. A principled method that exposes interpretable trade-off controls gives operators something to actually tune.
The researchers' conclusion — that a math-based approach beats one built on rules of thumb — sounds obvious in retrospect, yet most deployed systems still pick the rule of thumb. That says more about the engineering shortcuts baked into early LLM infrastructure than it does about the difficulty of the math.