A new diagnostic benchmark exposes the specific ways language model agents break down when calling external tools — and the results are more varied than aggregate scores suggest.
Researchers behind ToolFailBench ran 19 models through 1,000 tasks spanning finance, medicine, law, cybersecurity, and real estate. The benchmark separates tool-required tasks — where the model must trust a tool's returned value rather than guess — from control tasks where tools are available but the right move is to answer directly. Each interaction gets labeled across four failure modes: Tool-Skip, Result-Ignore, Output-Fabrication, and Unnecessary-Tool-Use. The best-performing model hit an 86.33% Clean Tool-Use Rate, meaning even the leader fumbles roughly one in seven tool interactions.
The more telling finding is that models with similar overall scores fail in entirely different ways. Llama-3.1 models show an "Always-Call" pattern — reaching for tools even when they aren't needed — while Llama-3.1-70B and Qwen2.5-72B, matched closely on parameter count, differ by 89 percentage points on control-task accuracy. That gap makes aggregate leaderboard comparisons nearly meaningless for anyone deploying agents in high-stakes domains.
Tool calling is the connective tissue of most production agent systems today, so a benchmark that can distinguish "model ignored the tool output" from "model called the right tool correctly" is genuinely useful — assuming labs treat diagnostic granularity as a target rather than a footnote.