A new benchmark says AI agents can barely reproduce machine learning research, even when the code is handed to them.
Researchers built RECLAIM, a benchmark of 100 papers from NeurIPS 2025, designed to be rebuilt each year with fresh papers. Before testing an agent, the team locks in the specific result to reproduce, the criteria for success, and a GPU-hour budget in advance. Papers are sorted into three tiers based on what their authors actually released: Run-tier papers ship code, data, and trained weights; Retrain-tier papers withhold weights, so the agent has to train the model itself; Reimplement-tier papers withhold code entirely, forcing the agent to write it from scratch. A separate language model grades each attempt by checking logs and outputs, not by trusting the agent's own account of what happened.
The results are blunt: the best agent tested reproduced only 41% of Run-tier papers, where everything was handed to it, dropping to 27% for Retrain and just 15% for Reimplement. Failed runs burned only 29% of their allotted budget on average, meaning agents mostly gave up rather than ran out of resources. The single most common failure, seen in 63 of 400 runs, was writing a method without ever checking it against the paper's own reported numbers.
That's a useful gut check for anyone picturing autonomous agents churning through research at scale: right now, most struggle to faithfully copy a known result, let alone discover a new one.