A new paper argues that the explanations AI chatbots give for their own answers can sound entirely convincing without actually being true - and that this might not matter as much as you'd think.
The paper, posted to arXiv, looks at "self-explanations": the plain-language justifications large language models produce when asked why they answered a certain way. The authors say these explanations are often highly plausible, meaning they read as sensible and convincing to a person. But plausibility isn't the same as faithfulness - whether the explanation actually describes the reasoning the model used to reach its answer. Current testing methods, they argue, don't do a good job telling those two things apart, so they lay out practical guidelines for evaluating both, plus a third measure: whether an explanation is actionable enough to help someone make a decision.
This matters because self-explanations are already being used as a stand-in for real interpretability, letting companies point to a chatbot's stated reasoning as evidence their AI is explainable, without proving that reasoning matches what the model actually computed. The paper's more interesting claim is that an explanation can still be genuinely useful even when it doesn't accurately reflect the model's internals.
That's a more honest framing than most "explainable AI" marketing offers, but it also means the bar for trusting an AI's stated reasoning just got a bit more complicated than simply asking is it true.