Real-time voice AI has a failure mode nobody warned you about: it doesn't slow down under pressure, it falls off a cliff.
Researchers tested full-duplex voice models — the kind powering systems like Moshi, MiniCPM-o, and Qwen-Omni — and found a nasty property in how they handle sustained load. Each active session pins its key-value cache in GPU memory for the entire conversation, and that cache grows without bound. When enough sessions stack up, the serving engine doesn't degrade gracefully; it stalls completely, jumping from milliseconds per frame to a frozen engine in a single step. Worse, standard latency and deadline-miss metrics look healthy right up until the collapse. In testing, 14 out of 20 five-minute runs crashed — and whether a given run survived was essentially random.
The fix the team proposes, called Metronome, is conceptually simple: cap how much resident state each session can hold. Bounding the KV window eliminated the collapse entirely (0 out of 20 runs failed) and turned per-frame latency into a reliable signal that actually reflects system load. That honest signal lets an admission controller find the real concurrency ceiling instead of over-admitting sessions until the engine hits the wall. A small number of "attention-sink" tokens are kept pinned to preserve coherence in free-running generation, and the researchers report the window imposes no quality cost in turn-based decoding.
The broader point here is architectural debt hiding in plain sight. The voice AI stack borrowed serving assumptions from text models, where sessions end and memory is reclaimed. Real-time interaction models break that assumption, and operators running them at scale may have no idea how close to the edge they are until it's too late.