LLM explanations look plausible but routinely fail to account for how the model actually reached its answer.
Researchers have introduced a framework called self-consistent sufficiency — and a metric, SCSuff — to test whether the explanations LLMs generate, such as chain-of-thought reasoning or post-hoc rationales, genuinely capture the model's output-generating process. The approach works by using the model itself to produce alternative inputs consistent with a given explanation, then measuring how much information the explanation actually carries. Across experiments, LLM explanations came up short: they were generally insufficient, and sufficiency showed only weak correlation with model size, accuracy, or output entropy. The team also found that hidden-state representations at the final token can predict SCSuff scores, which opens a path toward detecting and improving explanations internally.
This matters because high-stakes deployments — medical triage, legal analysis, content moderation — increasingly lean on these explanations as a form of accountability. If the rationale a model produces is decorative rather than diagnostic, the audit trail collapses. The research formalizes what many practitioners have suspected informally: that a convincing explanation and a correct one are not the same thing.
The findings sit alongside a growing body of work questioning whether chain-of-thought reasoning reflects actual model computation or is itself a learned output pattern — plausible prose that post-rationalizes rather than explains.