Researchers built a system that tests whether AI explanations actually predict future model behavior, and found that many popular explanation methods do not.
A team introduced CHIVE (Counterfactual Hypothesis Investigation Via Edits), an automated pipeline that hunts for surprising behavior in large language models, then probes it with counterfactual prompt edits to see whether a proposed explanation holds up on related inputs. The method generated thousands of explanations paired with counterfactual evidence, detailed in a paper posted August 18, 2026 on arXiv. The researchers tested whether standard LLM interpretability techniques help an AI agent predict how a model will respond to a tweaked version of a prompt. They also used CHIVE's output as training data for a separate model.
None of the interpretability techniques tested produced any measurable improvement in predicting counterfactual behavior, a null result for methods often marketed as windows into model reasoning. Training a model directly on CHIVE-generated counterfactual experiments, however, did generalize to new, out-of-distribution settings, suggesting the raw experimental data may be more useful than the explanations meant to summarize it.
That is a pointed challenge to interpretability and chain-of-thought faithfulness research, a field regularly cited as evidence that AI systems can be understood and trusted. If an explanation cannot predict what a model will actually do, it is decoration, not insight.