Nightjar is a new inference framework that solves a long-standing tension in large language model serving: the technique that speeds up generation under light traffic can become a drag when load spikes.
Research posted on arXiv describes Nightjar as an adaptive wrapper around speculative decoding. That technique works by running a smaller draft model to predict tokens ahead of time, letting the main model verify batches in parallel and cutting the number of sequential generation steps. Under heavy load, though, verification overhead grows and keeping the draft model in GPU memory shrinks available batch capacity, often reversing any gains. Nightjar adds a planner - drawing on multi-armed bandit methods from reinforcement learning - that continuously monitors whether speculation is still worth running. When it is not, the system shuts it off and offloads the draft model to CPU memory, freeing GPU headroom for larger batches.
Most production deployments see traffic that swings throughout the day, and a serving stack tuned for idle-hour efficiency typically stumbles under peak load. Nightjar's reported results - up to 14.76% higher throughput over standard speculative decoding and up to 20.18% lower latency - suggest a dynamic control loop can shift the efficiency curve across the full operating range rather than locking in gains at just one end.
The obvious caveat: this is a research paper, not a shipping product. Benchmark gains measured under controlled dynamic request patterns may not survive contact with real production traffic, and the planner itself adds overhead the authors treat as negligible.