Researchers built a benchmark that pits AI web agents against arXiv's paper library, and most of them stumble.
WebArxiv is a set of 510 tasks built on frozen, static snapshots of arXiv's site, so results don't shift when the site's layout changes. Each task has one deterministic correct answer, and the tasks go beyond simple lookups: agents have to retrieve papers matching multiple constraints, pull specific details out of paper content, and compare findings across papers. The researchers tested several foundation-model-based web agents against this benchmark. Behavioral analysis found that these agents over-rely on their fixed interaction histories, which leads to incomplete or repetitive reasoning.
Most web-agent benchmarks test general browsing, filling shopping carts, clicking through menus, not the kind of multi-step research work that knowledge workers actually do. WebArxiv's static-snapshot design also solves the reproducibility problem that plagues live-site benchmarks, where a redesign can quietly invalidate a year of comparisons. The failure mode it surfaces, agents leaning on stale conversation history instead of re-checking the page, is a structural weakness in how these agents are built, not a quirk specific to arXiv.
That matters beyond academic search. Any agent set loose on multi-step tasks, like filing expenses or cross-referencing documents across tabs, inherits the same risk of running on outdated context instead of the page actually in front of it, and that is exactly the kind of silent failure that is hardest to catch in production. The researchers' fix, a lightweight dynamic-memory mechanism for pulling in relevant context on demand, is a patch rather than a cure, and the benchmark and code are still hosted at an anonymized repository rather than a public release.