Running large language models is expensive, and a new paper argues the industry is wasting GPU time it does not need to waste.
Researchers surveyed existing LLM inference systems and found a common gap: almost none have formal models to check whether they are actually squeezing the most out of their hardware. Their fix draws from an unlikely source — database management systems, which have decades of theory behind cache replacement policies. The team built simulation tooling that lets developers test inference configurations without spinning up real GPUs, then proposed a cache replacement policy borrowed from the database world that plugs into existing schedulers. On real online workloads, the approach meaningfully cut GPU hours consumed during both development and live serving.
The broader point is about engineering culture. LLM inference research has mostly focused on tuning parameters within a given system, not on questioning whether the system's foundations are sound. Treating inference infrastructure like a database problem — with theoretical performance bounds and formal cache analysis — opens a different class of optimizations that the field has largely ignored.
The authors also note they submitted this work to a database venue in November 2025, a small but telling detail: the most useful ideas for AI infrastructure may be sitting in fields that AI researchers rarely read.