Coding agents can now replicate computational claims in machine learning papers - and a new workflow is designed to make sure they actually check their work.
Researchers introduced Paper-replication, a structured workflow that turns each claim in a scientific paper into a tracked target. The agent reconstructs the paper's method, runs experiments, links outputs to the original claims, and must pass validation checks before it can call the job done. Tested across twelve independent runs on four scientific machine learning papers, every run passed the completion gate and all 158 recorded targets were matched with documented evidence. The key design choice: completion depends on workspace evidence, not on whatever the agent writes in its final message.
That last part matters more than it sounds. Most coding agents are evaluated on whether they produce plausible-looking output, which turns out to be a low bar - a confident summary and a passing vibe are easy to fake. Paper-replication shifts the burden of proof to verifiable artifacts, which is a more honest standard for scientific replication. If this approach scales, it could put real pressure on the reproducibility problem that has dogged machine learning research for years.
The researchers also noted that repeated runs varied in how papers were divided into targets, numerical fidelity, and replication time - a reminder that "passed" is not the same as "identical", and that automated replication is still a probabilistic process, not a rubber stamp.