Researchers built a benchmark that watches how AI agents evolve, not just where they end up.
The new benchmark, called EvoPathBench, targets self-evolving agents, systems that turn feedback from their interactions into memories or skills that shape future decisions. Instead of only scoring the agent's final output, EvoPathBench freezes the base model and tools, then checks a fixed target capability at successive checkpoints as the agent's self-generated artifacts change over time. Using public trading data and calibrated trajectories, it tests three things: whether an agent generalizes to new but similar tasks, whether it keeps earlier skills after learning something unrelated, and whether it updates its own rules when new evidence shows up. Across the methods tested, gains on similar unseen tasks often weakened once the task distribution shifted, and no method reliably adapted its rules to new evidence.
Most agent benchmarks grade the finish line, which hides how an agent got there and whether its self-updates actually helped. This work shows agents can generate candidate updates with real, measurable gains on held-out tasks, but the step where the agent picks which updates to keep consistently fails to capture that potential.
Letting an agent write its own memory doesn't mean it's writing anything worth keeping.