A new benchmark says most LLM inference speedups are overhyped, and only a handful reliably pay off.
The paper, an arXiv preprint (arXiv:2609.17863), ran 54 configurations of Qwen2.5-7B-Instruct on vLLM 0.12 across L4, A100, and H100 GPUs, then calibrated a simulator to match real measurements within 1.5 percent. It layered a quality check on top, testing FP16, 4-bit AWQ, FP8 weights, and FP8 KV cache on 200 GSM8K math questions. Combining optimization methods reached the cost-quality-latency frontier far more often than using any single trick alone: 9 of 15 combos made the cut versus 9 of 21 solo methods. Aggressive 4-bit quantization looked great on paper, cutting per-token latency to a third of baseline on L4 GPUs, until the accuracy check knocked off nearly 6 percentage points of GSM8K accuracy, just missing the study's 95 percent quality floor.
That gap is the real finding here. FP8 weights, a milder form of quantization, kept 99.4 percent of baseline accuracy while still cutting latency by roughly 35 to 40 percent, and won three of four deployment scenarios the study tested. A naive FP8 key-value cache, by contrast, kept full throughput and got every single test question wrong, fast and useless. Speculative decoding, often pitched as a free lunch, added no benefit at all on this stack.
None of this settles the debate for good; it is one model family on one serving stack. But it is a useful corrective for anyone reading GPU vendor slide decks that show throughput charts and nothing else. H100 wins on raw latency, A100 wins on cost at $0.106 per million tokens, and the fastest option on any leaderboard might just be the one nobody checked for right answers.