A new benchmark finds that web agents with nearly identical task-completion rates fail in entirely different ways, a gap that current evaluations miss entirely.
Researchers built WebStep, a collection of 1,800 task instances that pairs a standard browser interface with a semantic state tracker running silently in the background. Instead of only recording whether the agent finished the job, WebStep logs every intermediate state and transition, essentially a flight recorder for AI browsing sessions. That process data revealed something pass/fail scores obscure: three leading web agents all completed tasks at roughly the same 31-33% rate, yet their failure profiles were almost opposite. On a housing-search task set, OpenAI's CUA outperformed Qwen3.5 by 23.7 percentage points on committing actions but trailed by 15.6 points on filtering, the kind of specific diagnostic a binary outcome score cannot surface.
Most of the web-agent field is currently judged on benchmarks where you either complete the task or you don't. That's useful for ranking models on a leaderboard but tells a development team almost nothing about what to actually fix. WebStep's process-level approach lets a researcher say "this agent executes well but explores poorly" rather than "this agent scores 32%", and the gap widens sharply as tasks get harder.
Whether the research community adopts process-level evaluation or keeps chasing terminal-success leaderboard numbers will say a lot about whether the field is serious about improving agents or just scoring them.