A new evaluation protocol for AI testing systems is built to shut up when it doesn't have the evidence to back up a verdict.
Researchers designed a three-part protocol - refuse, decompose, and refresh - meant to stop AI evaluation pipelines from issuing a clean pass or fail when the underlying data doesn't support one. They tested it on a simulator built around 24 policy components, three traffic-demand regimes, and two families of injected faults, checking results against a preregistered held-out set of 1,440 cases and 21,600 partition rows. Of 72 regime-component combinations, only 55 had enough clean reference data to be evaluated at all, and one of those lost its standing once real operating conditions ran. Of the 24 components tracked, just 20 had enough admitted evidence across regimes to be folded into the final false-admission count, which came back at zero.
That zero is less interesting than the refusals around it. Most benchmarks report one score and move on, which buries exactly the kind of gap this protocol is built to expose: the 17 regime-component pairings that got declined rather than measured, and the four components that didn't have enough admitted evidence across regimes to make it into that stable count. In closed-loop systems, where the policy under test also decides what gets observed, the difference between "we checked and it passed" and "we couldn't check" is not a technicality.
A drift-detection test made the stakes concrete: identical clean traffic triggered false alarms 15 out of 15 times in one regime, 0 out of 15 in another, and 14 out of 15 in the third, with only the middle regime matching the frozen detector reference. A single fixed threshold would have called the same clean signal three different things - which is a good argument for building the shrug into the scoring.