Most AI benchmark headlines are less rigorous than they look.
A newly published Python library called evalci takes a per-item results table from a language model evaluation run and returns a fully cited statistical claim - confidence interval, p-value, sample size, and all - in a single function call. Built on numpy, scipy, and pandas, it requires no additional dependencies and ships adapters for two widely used evaluation frameworks, lm-evaluation-harness and HELM. Every routine is cross-validated against an independent reference implementation rather than only against itself, which rules out a category of bugs where a method agrees with its own code but not with the math.
The motivation is blunt: the standard practice of comparing two models by their raw accuracy scores, with no test of whether the gap exceeds sampling noise, routinely overstates confidence. On benchmarks with a few thousand items and under temperature sampling - where a single model can vary from run to run by more than the margin separating it from a rival - a 1- or 2-point lead is often meaningless. The authors demonstrate this concretely by re-analyzing a public nine-model MMLU comparison and finding that 3 of the 8 adjacent leaderboard-rank gaps fail to reach significance once the 36 implied pairwise comparisons are properly corrected for.
This matters because leaderboard rankings drive real decisions - which model a team adopts, which lab attracts funding, which research direction gets pursued. A statistical artifact dressed as a benchmark win is not a minor rounding error.
The statistical tools evalci packages - paired permutation tests, clustered standard errors, multiple-comparison correction - have existed for decades in other fields. The gap was always packaging, not knowledge.