An AI agent built to run experiments on a video recommendation system caught its own team's earlier mistake, twice, before either conclusion could ship.
The system, called EvoPilot, runs human-gated 'autoresearch' loops: LLM agents propose, implement, and test changes to a live retrieval model, in this case the engine behind Video Deep Dive, a video-discovery product built on an hourly-refreshed index of hundreds of millions of clips. Over a 37-day campaign covering seven research directions, the agents revisited an 'interaction head' that earlier manual experiments had already dismissed as useless. A first, less rigorous autoresearch pass blamed that head for a 22-percentage-point drop in offline hit rate. EvoPilot's verification step traced the real cause to a pre-existing evaluator bug that compared results at mismatched depths of 3,000 and 600 items.
Once the bug was fixed, the same head actually improved offline hit rate by 3.20 percentage points, and a seven-day randomized online test measured a 0.66% relative lift in the product's core quality metric. That is the notable result here: not that agents can run experiments faster, but that a documented verification layer caught both a false negative and a false positive in the same investigation. The paper also notes the system recovered an interrupted round from durable state and reused artifacts to save roughly five GPU-hours, mundane but real efficiency gains.
Every lab racing to automate its own research pipeline should sit with the mismatched-evaluator detail. A faster agent that trusts a broken pipeline does not find answers any quicker. It just breaks things at a higher velocity.