AI agent benchmarks are a mess, and a new research framework wants agents themselves to clean it up.
A team of researchers has published AgentBeats, a system where judge agents evaluate subject agents through standardized protocols rather than bespoke harnesses. The framework uses two open protocols — A2A for task management and MCP for tool access — both of which have been gaining traction as cross-lab standards. To stress-test the approach, the team ran a five-month open competition that drew 298 judge agents across 12 categories and 467 subject agents submitted by independent participants. A separate coding-focused case study found that agentified evaluation matched the public benchmarking record while also surfacing head-to-head comparisons that conventional harnesses had left blank.
The standardization angle is the real value here. Most benchmarks currently define two separate interfaces — one for the benchmark, one for the agent — which means each new test suite demands custom integration work and produces results that are nearly impossible to compare fairly across different research groups. Collapsing that to one interface, built on protocols the industry is already converging on, lowers the cost of running reproducible comparisons without requiring everyone to defer to a single central scorekeeper.
The catch: judge agents are still models, and the track record of LLMs grading LLMs is complicated enough that "preserves fidelity with the public record" is a claim that probably deserves watching over the next few months of real-world use.