Nvidia's Nemotron lab published a compressed large language model that fits more concurrent users onto the same hardware without gutting benchmark scores.
The model, Nemotron-Labs-3-Puzzle-75B-A9B, is a slimmed-down version of Nemotron-3-Super built specifically for interactive serving. On a single 8xB200 node it reaches roughly 2x the server throughput of its parent at equivalent user load. Swap to a single H100 GPU and ultra-long 1M-token context, and concurrency jumps from one simultaneous request to eight. The compression pipeline runs the Iterative Puzzle framework alongside knowledge distillation, reinforcement learning, quantization, and a Multi-Token Prediction head — jointly tuning heterogeneous MoE pruning, active parameter budget, and Mamba pruning in one pass rather than treating each as a separate knob.
The throughput numbers matter because inference cost is where most AI deployment budgets bleed out. Serving twice as many users on the same node, or eight times the long-context requests on a cheaper GPU, is not a research curiosity — it is a real reduction in per-query infrastructure spend. The Mamba pruning angle is also worth watching: hybrid architectures that blend attention and state-space layers are still rare enough that a public compression recipe for them is genuinely useful to the field.
Nvidia is not the only lab working this problem — Meta, Mistral, and several startups are all chasing cheaper inference — but the explicit 1M-token concurrency claim on a single H100 is a specific, falsifiable benchmark rather than the usual vague efficiency marketing.