AI infrastructure researchers have designed a way for inference servers to share spare capacity instead of sitting idle.
Every AI chat or agent request breaks into two jobs: reading the prompt, called prefill, and generating the reply, called decode. Most large-scale AI systems run these jobs on separate, dedicated machines because it's more efficient, but that split is usually fixed in advance. A new system called Crossflow, described in a paper published on arXiv, instead lets each machine lease out spare capacity to the other job on the fly, without reassigning which job it's nominally built for. The researchers found demand for each job is anything but steady: it swings up to 4.7 times above average within a single minute, and by a median of 24.5 times across a day in real agentic AI traffic, the multi-step, tool-using tasks now common in AI agents.
That volatility is expensive either way. Sizing each pool of machines for peak demand leaves up to 17 percent of a cluster sitting idle, and sizing it lower just turns the same swings into slower replies for users. Crossflow's approach recovered much of that waste in testing, lifting throughput by 16.2 to 17.4 percent on average and as much as 43.4 percent under heavy load, while also cutting the wait for a reply's first word at every load level tested.
It's a research result, not a shipped product, and the real test will be whether any cloud provider bothers rebuilding its serving stack around it.