A new benchmark says popular AI explainability tools may not be explaining much at all.
Researchers built a framework that creates synthetic datasets with built-in, known answers about which inputs actually matter to a model's decision. Using controlled interventions across binary images, tabular data, and time series, they could check explanations against a real ground truth instead of guessing. They tested nine widely used explainable AI (XAI) methods against that standard and found significant limitations in how well the methods captured what the underlying models were actually doing.
That distinction matters because most XAI evaluation today just checks fidelity - whether an explanation matches a model's output - not whether it accurately describes the model's decision process. Two explanations can score identically on fidelity while telling completely different stories about why a model made a call. That gap is exactly what makes explainability claims hard to trust in high-stakes uses like medical diagnosis or loan approvals.
Explainability tools have always asked us to take their word for it. This framework is a first step toward making them show their work instead.