A single benchmark score for AI root-cause analysis tools can name the wrong winner for your specific infrastructure.
Researchers audited three public root-cause analysis benchmark families — OpenRCA, RCAEval, and PetShop — covering 11 subsystems and 778 matched scoring units. They compared four methods: BARO, a CD-1min adapter, max-|Z|, and per-service alert-count. All six pairwise comparisons showed subsystem-level effects pointing in both directions, and case-level interaction tests rejected exchangeability in five of six pairs. When the team ran a leave-one-system-out selection exercise, the method chosen by pooled scores turned out to be the lower-scoring option on up to 5 of the 11 held-out subsystems — with regret reaching 24.8 percentage points on the RCAEval Sock-Shop benchmark.
The finding matters because engineers routinely treat a pooled leaderboard winner as a deployment recommendation. That reading is statistically unsupported: a method that ranks first overall can rank last on the subsystem that actually resembles your environment. The researchers released a 320-line audit module that recomputes per-subsystem stability checks alongside pooled scores, giving teams a concrete way to pressure-test any benchmark table before acting on it.
Benchmark literacy has been a slow-burning problem across ML for years — from image classifiers gaming ImageNet to LLMs saturating reasoning tests. Root-cause analysis is just the latest domain where a tidy aggregate number turns out to be marketing dressed as measurement.