A new study puts four open-source LLM routers through identical benchmarks - and finds most of them aren't really routing at all.
The paper, "Task- and Session-Level Model Routing: A Common-Interface Hybrid Evaluation of Four Open-Source Routers Across Four Benchmarks" (arXiv:2608.14641), tested four router implementations against a locked matrix of 2,610 candidate outcomes drawn from 290 tasks across RouterBench, BFCL v4, tau2-bench, and WebArena. Three of the routers issued constant or near-constant tier assignments no matter what was asked. Only vLLM Semantic Router actually varied its picks based on prompt content - and even it posted the highest success rate on none of the four benchmarks. A simple always-pick-the-middle-tier baseline matched one router, Aurelio, exactly on three benchmarks and within 0.003 on the fourth.
That matters if you're paying a premium for "smart routing" to cut inference costs. The study's task-level tests found no measurable task-specific advantage for vLLM's router over a content-blind allocation matched to the same tier distribution, with equivalence established only on WebArena. In plain terms: the gains routers claim track which tier gets picked how often, not genuine judgment about which model fits which task.
Routing was supposed to be the free lunch of multi-model agent stacks - cheap tasks to small models, hard ones to big ones. This benchmark suggests most routers are just flipping a weighted coin and calling it strategy.