A new benchmark asks a harder question than whether an AI agent can complete a task — it asks whether the agent got better at completing tasks because of what it did before.
Researchers introduced EvoAgentBench, a framework designed to evaluate agent self-evolution through what they call Ability-guided transfer. The benchmark extracts reusable procedures from agent execution traces — think search strategies, debugging routines, verification steps — and builds graphs linking tasks that share procedural overlap. It then tests whether an agent that learned a procedure on one task can apply it to a related one. The dataset covers four domains: web research, algorithmic reasoning, software engineering, and knowledge work, split across 528 training and 267 test tasks.
The distinction matters because existing benchmarks measure the wrong thing. Agent benchmarks score single-episode task performance; memory benchmarks test whether an agent recalls facts. Neither asks whether experience compounds into reusable skill. That gap means the field has been optimizing for one-shot problem-solving while quietly ignoring whether agents actually improve over time — which is what "self-evolution" would have to mean in practice.
The results are a measured reality check: curated Ability content transferred reliably across model families, but no current automatic method sustained positive gain across all settings. In other words, hand-selected experience works, but autonomous experience extraction does not consistently. That is a useful finding, even if it is not a flattering one for the labs promoting agentic AI as a near-term unlock. The benchmark is publicly available, so anyone claiming their agent "learns" now has a harder test to pass.