A new paper proposes routing all three major inference efficiency decisions through a single learned controller — and claims the results beat handling each one in isolation.
Most large language models that cut inference costs do so by picking a lane: Mixture-of-Experts sparsifies feed-forward layers, Mixture-of-Depths skips transformer blocks entirely, and KV-cache quantization shrinks attention memory. TriRoute, from researchers posting to arXiv, argues these three decisions are deeply coupled. A token rare enough to warrant full attention probably also needs high-precision memory storage, regardless of which expert handles it. Their lightweight controller emits a coordinated policy for every token at every layer: how much attention to apply, which experts to activate, and at what bit-width to cache. Training the system jointly exposed a problem they call a "routing-collapse cascade" — failure on one axis cascades to the others — which they address with per-axis normalization and a coupling-aware loss function.
The efficiency gains matter because they target the tail cases that simpler compression schemes quietly sacrifice. On decoder-only models between 160M and 1.3B parameters, TriRoute reportedly outperforms the best independent combination of all three techniques at matched compute and memory budgets, while holding up better on rare entities, code, and arithmetic — exactly the inputs where perplexity scores look fine but real-world outputs degrade. The controller also develops legible behavior: it allocates full attention and high-precision cache to sentence starts, rare subwords, and named entities, while routing function words cheaply.
No code or production integration is announced yet, and the experiments top out at 1.3B parameters — a scale well below where inference costs become existential for most labs.