Picking the right AI agent for a task is harder than it sounds — and now there's a formal benchmark to measure it.
Researchers published a benchmark derived from WildChat, a large dataset of real user conversations, containing 3,000 prompts mapped against a fixed catalog of 12 agents. The core problem they're testing is set-valued prediction: a single query might need one agent, or it might need three, and over-selecting runs up execution costs. Their evaluation protocol layers several metrics — Precision, Recall, F1, Jaccard, Exact Match — alongside latency and a cost-aware simulation that uses ordinal agent-cost tiers to score constrained routing decisions.
The findings matter because zero-shot LLM routing, the obvious default for anyone building a multi-agent system today, finishes near the bottom of the rankings. Supervised approaches — particularly a fine-tuned encoder — substantially outperform it on raw accuracy, while a simpler linear multilabel classifier holds up well as a practical baseline. Adding a weighted post-scoring layer called WAR on top of the encoder produced the best cost-constrained results.
This is the kind of unglamorous infrastructure work that determines whether multi-agent systems are actually useful or just expensive. Most coverage of AI agents focuses on capabilities; this focuses on dispatch, which is where real-world deployments quietly break down.