A paper posted to arXiv, arXiv:2601.21351, gives LLM infrastructure teams a formula for one of inference serving's most expensive guessing games: how many GPUs to spend on memory versus compute.
The paper studies attention-FFN disaggregation, an architecture that splits LLM decoding into two separate jobs. Attention workers hold the growing KV cache and handle state-heavy lookups; FFN workers do the stateless, compute-heavy math; the two communicate every step. That split lets operators scale memory and compute independently, but performance hinges on getting the ratio of attention to FFN workers right, since a bad ratio leaves workers idle waiting on the slowest step. The authors model the randomness in prompt lengths, decode lengths, and cross-worker synchronization, then derive a closed-form rule for the optimal ratio.
Companies running inference at scale usually tune these ratios by trial and error, burning GPU-hours on test runs or over-provisioning as insurance. A formula that predicts the right split from a single measurable workload statistic could turn that guesswork into a calculation, which matters when idle GPU time is a direct cost.
Worth noting: the validation here comes from a trace-calibrated simulator that matched the true optimum within 10 percent, not a live production fleet, so the real test is whether the math holds up against the messiness of an actual serving cluster.