Current AI agent benchmarks are making models look better than they are.
Researchers published AgentGym2, a new evaluation framework designed to test large language model agents under conditions that more closely resemble actual deployments. Unlike most existing benchmarks, it does not hand agents pre-packaged tool interfaces or clean, fully specified inputs. Instead, agents must discover tools through exploration, compose them for tasks they haven't seen before, and stay on track when information is incomplete or noisy. The paper tested 15 proprietary and open-source models and found that even top-ranked systems struggled significantly — including Google's Gemini and what the benchmark authors labeled as OpenAI's top-tier model, though that specific labeling could not be independently verified against publicly documented release lineups.
The gap matters because the AI industry has been benchmarking its way to optimism. When evaluations assume clean inputs and pre-wired tools, they measure a laboratory version of the problem, not the one companies actually encounter when they ship agents into production. AgentGym2's results suggest the real-world performance ceiling is considerably lower than leaderboards imply.
This follows a familiar pattern: a new, harder benchmark arrives, top models stumble, labs announce they're working on it, and the cycle restarts. The more interesting question is whether evaluation frameworks like this one will influence how models are trained, or just how they're marketed.