An AI research team has built a web crawling agent that knows when it has gotten lost.
Researchers from Yonsei University released BaRA, short for BFS-and-Reflection Agent, a framework designed to make LLM-based web agents more reliable at site-level data collection. The system combines bounded breadth-first search — a structured way of exploring a website layer by layer — with a history-based self-reflection loop that lets the agent notice and correct its own gaps. The team tested BaRA against 50 synthetic websites with known ground-truth data, plus three live public sites with cluttered or dynamic layouts. It outperformed three existing approaches — Pure LLM, SeeAct-Vision, and Browser-use — with the biggest gains in recovering downloadable images and video, which are notoriously easy for agents to fumble.
The problem BaRA targets is real and underappreciated: LLM agents sent to collect data from websites routinely skip pages, return partial results, or surface media URLs that point to nothing downloadable. That makes them unreliable for any pipeline that actually needs complete datasets. BaRA's fix is architectural — bake exploration discipline and self-correction into the agent loop itself, rather than hoping the model figures it out.
The work lands as agentic web tools proliferate without much scrutiny of their failure modes. Browser-use, one of the baselines BaRA beats, drew attention earlier this year as a general-purpose browser agent; the fact that a purpose-built crawling framework outperforms it on structured collection tasks suggests the general-purpose framing may be covering up some quiet gaps. Code is available on GitHub, which at least makes the comparison reproducible.