Some of AI's most confident wrong answers are not glitches. They are stable.
A new arXiv paper studies "stable miscalibration" in large language models, cases where a confident wrong answer holds steady even when the input is slightly perturbed, rather than wobbling like a shaky guess. The researchers built two diagnostics: an audit score that flags domains with mismatched confidence and forced-answer overconfidence, and an internal probe that tracks how much a model's hidden states shift under perturbation. Testing across a multi-domain binary factual set, they found that self-critical prompting, where a model is asked to reconsider or abstain, consistently reduced that internal hidden-state sensitivity across layers in three open-weight models. But the audit score's link to actual decision-loss reduction was weaker than direct labeled baselines, and the paper found no clear evidence that these overconfident errors were internally shakier than correct answers.
That last point matters more than it sounds. Most efforts to fix AI hallucinations assume wrong answers are computationally fragile, that nudging the model should crack the error loose. This research suggests some wrong answers are just as internally settled as correct ones, meaning confidence, whether read by a human or measured inside the model, is a poor proxy for correctness.
Self-critique prompting still changed how settled a model's internal state looked. It just did not reliably change whether the model was right. Stability and truth, it turns out, are not the same thing, and this paper is among the first to measure the gap between them directly.