AI agent benchmarks have a methodology problem, and a new suite aims to name it.
Researchers have released an executable benchmarking suite that connects three established agent evaluation environments — WebArena Verified, a SWE-Gym slice compatible with SWE-bench, and MiniWoB++ — under a shared "evidence-admission contract." That contract draws a hard line between results that count as paper-facing evidence and rows that exist only for diagnostics, preflight checks, or onboarding. It also requires explicit declaration of what is driving action generation, something most benchmark reports quietly omit. The suite captures latency, invalid-action rates, patch-generation cost, and verifier metadata in one auditable record.
The distinction matters more than it sounds. The researchers ran a controller study on WebArena Verified and found that a clean-baseline evaluation and a medium live-stressed evaluation picked different winning controller variants under the same workload and admission rules — meaning the apparent best system changes depending on how stress is applied. That is exactly the kind of variance current benchmark reports routinely paper over.
The broader problem is that AI agent evaluation has scaled faster than the tooling meant to keep it honest. Labs publish scores on SWE-bench or WebArena as if they are comparable, but the underlying drivers, replay policies, and evidence gates often differ in ways no one discloses.
The authors are careful to scope this as benchmarking infrastructure, not a new agent, model ranking, or SWE-bench solver — a refreshingly modest framing in a field where every release tries to be a leaderboard topper.