LLMs are confident for the wrong reasons — and a new study explains why.
Researchers tested four non-reasoning and four reasoning models using a two-stage setup borrowed from neuroscience: a model answers, reports its confidence, then decides whether to deliver the answer or abstain. Verbal confidence scores predicted the commit-or-abstain decision far better than they predicted whether the answer was actually correct. Token log-probabilities, by contrast, tracked correctness much more closely. When the researchers stripped out the variance verbal confidence shared with log-probabilities, the leftover signal stayed locked to commitment behavior while its connection to correctness dropped to near chance. Mechanistic probes of Gemma 3 and 4 reinforced the finding: the internal state that drives verbal confidence is organized around the abstention decision, not correctness, with the two sitting in roughly orthogonal directions in activation space.
This matters because many AI reliability tools treat verbal confidence as a proxy for accuracy — a shortcut that this research suggests is largely wrong. If a model says "I'm fairly confident" it is describing an internal readiness-to-commit state, not an honest probability estimate. Calibration methods and human oversight workflows built around verbal reports may be measuring the wrong thing entirely.
The field has long known that LLMs can hallucinate with conviction, but this gives that observation a more precise mechanism. The practical fix implied by the paper is unglamorous: route reliability judgments through log-probabilities, not the model's own words about itself.