Larger language models are not just wrong sometimes — they are wrong with increasing confidence, and a new paper explains the geometry of why.
Researchers studying how language models store and retrieve knowledge identified two distinct failure modes. In the first, a model has learned a fact in its weights but the context window feeds it conflicting information — the two sources fight, and the model produces confident output anyway. In the second, the model was never trained on the fact at all, so its hidden state simply drifts with no anchor. The key finding: the frozen output head used for next-token prediction cannot tell these cases apart. It fires confidently in both. The researchers verified this on a controlled synthetic task using LoRA adapters, then confirmed it held on natural-language queries from a pretrained model without any fine-tuning.
What makes this matter is the scaling-law result buried in the paper. The fraction of confident hallucinations follows an exponential relationship with model scale — meaning as overall error rates fall, the errors that remain become harder to catch because they look exactly like correct answers. Output-based monitoring, the dominant approach in most production guardrails today, is structurally blind to this. The hidden states, by contrast, do encode epistemic state reliably. A metric the researchers call geometric margin — measuring how close the hidden state is to a memorized attractor basin — separates real recall from hallucination far more cleanly than output entropy, with zero false refusals.
The implication is uncomfortable for anyone selling confidence scores as a hallucination fix: the signal is there, inside the model, but the architecture's final layer actively discards it. Building detectors that tap hidden states rather than token probabilities is not a new idea, but this paper gives it a cleaner theoretical foundation than most prior work.