Ask a chatbot how confident it is in an answer, and it turns out that number isn't improvised on the spot.
A new interpretability study traces exactly how large language models generate the confidence scores they report when prompted to rate their own answers. Researchers tested Gemma 3 27B on trivia, math, and general-knowledge benchmarks, plus Qwen 2.5 7B and the reasoning model Magistral Small 24B. Using techniques like activation steering, patching, and attention blocking, they found confidence signals form immediately after the model writes its answer, get pulled from the answer's own tokens, and are cached at the first token slot after the response. When the model is later asked to state a confidence number, it retrieves that cached value rather than computing something fresh.
This matters because verbal confidence - a model literally saying "I'm 80% sure" - is one of the few tools researchers have for gauging whether a black-box model's output can be trusted. If that number were just a guess dressed up after the fact, it would be close to useless. Instead, the cached confidence signal explains meaningfully more of the variation in reported confidence than simple word-prediction probability (token log-probabilities) does, suggesting the model runs a genuine, if crude, self-check on answer quality.
None of this means the scores are accurate - a model can be confidently wrong, same as anyone. But it does mean that when an AI model states a confidence level, something resembling real evaluation happened under the hood, not a number pulled from thin air to sound rigorous.