Running AI chatbots on nearby cell towers instead of distant data centers is faster, but a new paper shows edge servers have been picking one bad tradeoff or another.
Researchers describe two ways to run large language model inference at the network edge. Autoregressive decoding generates one token at a time, which is slow but memory-cheap. Speculative decoding speeds things up by having a small model draft several tokens for the big model to check at once, but that requires loading a second model into memory. The new framework, called BALANCE, runs both modes simultaneously on one edge server, hosting a small and a large model together, and assigns each user to whichever mode fits their latency needs and the server's available memory. Because deciding the optimal mix is computationally NP-hard, the team built a faster approximate algorithm that splits the problem in two and still guarantees a solution within a known distance of optimal.
This matters because edge computing only works if it can serve lots of users on hardware with real memory limits, not the effectively unlimited pool of a cloud data center. Picking one decoding strategy for everyone means either wasting memory speculative decoding needs or making users wait through slow token-by-token generation. Letting a server run both at once, and shifting the mix based on demand, is a scheduling problem more than a modeling one - which is a more mundane but more immediately deployable fix than training a smaller model.
The paper reports throughput gains over using either method alone, though as with most inference-scheduling research, the real test is whether telecom operators building 5G and 6G edge infrastructure adopt something like it rather than shipping a simpler, less optimal system.