A research framework called AdaStop promises to make neural network testing cheaper by automatically deciding when to stop.
Most DNN testing methods rank inputs by how likely they are to expose model faults, then label them until a fixed budget runs out. The problem: nobody has a principled way to set that budget. Spend too little and you miss real failures; spend too much and you're paying for labels that reveal nothing new. AdaStop treats the stopping point as a cost-benefit calculation — labeling one more input costs c, finding a fault is worth v, so testing should continue only while the marginal fault-discovery rate stays above the ratio c/v. When that rate drops below the threshold, the framework stops.
The practical implication is significant for anyone paying per-label on a large test suite. Across multiple datasets, model architectures, and selection strategies, AdaStop's experiments show 65-84% of faults discovered using only 9-31% of the full labeling budget — meaning teams can cut label spend by roughly 70-91% without missing most of what matters.
The framing is tidy, but the threshold ratio c/v still requires someone to assign a dollar value to discovering a fault — a judgment call that varies wildly depending on whether the model drives a car or recommends a playlist. That caveat aside, cost-aware stopping is a genuinely underexplored corner of ML testing, and the numbers here are hard to dismiss.