A model can pass every accuracy, calibration and coverage test and still be quietly wrong in how it reasons.
Researchers behind a new arXiv paper prove what they call a separation theorem: two models, one reliable and one compromised, can be mathematically identical under every existing prediction-based certification method, including accuracy, calibration and conformal coverage, yet diverge arbitrarily in explanation fidelity and real-world behavior. Telling them apart is impossible by looking at outputs alone; it requires inspecting the model's decision mechanism itself. The authors propose a competence envelope, a framework that combines prediction certification with explanation certification into a single deployable trust criterion. Tested across diverse datasets and model classes, it surfaced failure modes that accuracy and calibration checks alone did not catch.
Most AI trust today, from regulatory sign-off to a vendor's accuracy claims, rests entirely on prediction-side numbers. This paper argues that's a category error: those numbers cannot distinguish a genuinely sound model from one that reaches the right answer through fragile or compromised reasoning. For systems judging who needs a hospital bed or whether a building is safe to enter, that blind spot is exactly where failures would stay invisible until conditions shift.
It's the AI equivalent of a student who aces every multiple-choice exam by memorizing answer keys rather than understanding the material, fine until the questions change.