Biomedical AI benchmarks have a credibility problem, and a new toolkit wants to fix it.
Researchers at AI-in-Health released BioMedArena, an open-source framework that separates biomedical agent evaluation into six discrete layers — benchmark loading, tool exposure, tool selection, harness mode, context management, and scoring. The toolkit ships with 166 biomedical benchmarks, 75 tools across nine functional categories, and six agent harnesses including the team's own Mutual-Evolve approach. Adding a new model, benchmark, or tool requires only a short provider adapter rather than weeks of custom engineering.
The reproducibility gap it addresses is genuine: the same AI backbone tested against the same benchmark has produced different accuracy numbers across papers simply because the surrounding scaffolding differed. That makes it nearly impossible to know whether one model actually outperforms another or just ran on a more favorable harness. BioMedArena's shared evaluation surface makes those comparisons meaningful for the first time at this scale.
Equipping BioMedArena's reference components on 12 backbone models pushed past prior state-of-the-art on all eight representative benchmarks tested, by 15.01 percentage points on average — a gap wide enough to suggest the harness was doing a lot of the heavy lifting in previous leaderboard results. Whether that improvement holds outside the curated benchmark set is the next question no one has answered yet.