A new arXiv paper puts a hard number on how much accuracy risk AI companies quietly accept when they compress data to serve more requests at once.
Large language model servers routinely compress the working memory they keep for each request, called serving state, to fit more users onto the same hardware. Today's systems adjust that compression on the fly based on load with no guarantee about how much it degrades answers, and the existing certified methods that do offer guarantees rely on a statistical budget that, per the paper, runs out on every long request in the production stack it tested. The researchers built a replacement, an anytime-valid statistical ledger that stays sound no matter how long a request runs, and checked it against 352,333 real admission decisions on live traffic. In a pre-registered test on held-out data, it cut the rate of falling back to slower, uncompressed computation by more than half, from 30 percent to 14 percent, while holding the risk guarantee constant.
Compression decisions in AI infrastructure are usually judged on throughput and cost, not on a stated, checkable risk. This work turns that tradeoff into a number a provider can price and audit instead of a knob tuned by feel, and it backs the math with 228 theorems verified in the Lean 4 proof assistant, a level of formal rigor more common in safety-critical software than in AI serving stacks.
It is one preprint tested on one internal stack, not an adopted standard, and the distance between a clean proof and what a cloud provider actually ships to production is usually where claims like this go quiet.