A research team has built a general-purpose tool to automate the replication of published scientific studies.
VERITAS takes a paper, a code repository, or both, then extracts the study's core claims, executes the methodology, and resolves problems that come up along the way. At the end, it returns an importance-weighted Replication Score, a severity-rated log of every fix it applied, and the patched codebase. The system is built around CLI coding agents and is designed to work across domains. The researchers tested it on 65 papers spanning computer science, social science, medicine, and astrophysics, and report state-of-the-art results on two benchmarks - CORE-Bench and ReplicationBench - against Claude Code baselines.
The replication crisis has been a slow-burning problem in science for over a decade, and AI-assisted publishing is making the math worse: more papers are arriving faster than human reviewers can check them. Most prior automation attempts were benchmark-specific tools that only ran inside their own pipelines, leaving no general solution. VERITAS at least proposes a framework that could work across disciplines without being retooled for each one.
Whether a pipeline that auto-patches code during replication is reproducing results or quietly laundering them is a question the authors acknowledge but do not fully answer - worth watching as the tool moves toward wider use.