Researchers have built a benchmark for the software scaffolding around AI agents, not the models themselves.
The tool, called A2E (Agent Auditing Engine), evaluates agent harnesses - the orchestration layers that let large language models call tools, plan multi-step tasks, and recover from errors. It uses a standardized task format called the Agent Task Protocol to plug in evaluation tasks across different harnesses, then automatically logs execution traces while the agent runs. Instead of just checking whether a task succeeded, A2E scores harnesses across multiple dimensions: how efficiently they execute, how well they use tools, how they plan, and how they recover when something breaks. Testing across model-harness combinations, the researchers found substantial performance swings depending on task type, with no single pairing winning consistently.
That last finding is the useful part. Most agent benchmarks treat the model as the variable that matters and the harness as plumbing. A2E's results suggest the plumbing matters just as much - a strong model paired with a mediocre harness can underperform a weaker model in better scaffolding, depending on what you're asking it to do. For teams building agent products, that reframes the buying decision: picking a foundation model is only half the job, and the framework wrapped around it needs its own evaluation.
It is a sign of where agent tooling actually is right now - not mature enough that harnesses are interchangeable, but popular enough that someone had to build an auditor to tell them apart. The code is open-source, which means the real test is whether harness developers use it to fix weaknesses or just cite favorable numbers in a launch post.