A new benchmark says the way you chop up an AI agent's toolbox matters more than which model you plug into it.
Researchers built MCP-GRANITE, an open-source benchmark for testing tool-interface granularity in agents that use the Model Context Protocol to call outside tools. It runs 81 multi-step tasks across 9 domains, each instantiated at four granularity levels, from many narrow single-purpose tools down to one do-everything tool. Nine locally deployed models, from 268 million to 20.9 billion parameters, were tested across 8,748 trials - three runs per model-scenario-granularity combination, which is how 9 models times 81 scenarios times 4 granularity levels (2,916 configurations) becomes 8,748 total trials. The team measured task completion, tool-selection accuracy, argument accuracy, latency, and resource use.
A four-tool setup won. It beat the finest-grained split by 16.4% on task completion and the single monolithic tool by 33.6%, while nearly doubling argument accuracy. Model size barely predicted task completion and mostly just predicted latency - a 3.2B-parameter model at the right granularity beat a 20.9B model at the wrong one.
For anyone shipping agents on constrained hardware, that's the more useful lever: before reaching for a bigger model, check whether the tool list is the actual bottleneck.