A research team has a new way to make neural network decisions explainable — one that sidesteps the main weakness of existing methods.
Most post-hoc explainability tools work by perturbing inputs: scramble a pixel, mask a word, then see how the model's output changes. The problem is those mutated inputs often fall outside the distribution the model was trained on, making the resulting explanations unreliable. The new framework, called Activation-Deactivation (AD), takes a different approach: instead of corrupting the input, it switches off the corresponding parts of the model itself during the forward pass. The researchers implemented this as ConvAD, a drop-in addition to any trained convolutional neural network that requires no retraining.
The distinction matters because explainability is no longer a research curiosity — regulators in the EU and elsewhere are starting to require that automated decisions be interpretable. If the explanations those systems produce are artifacts of out-of-distribution inputs rather than genuine model reasoning, they fail both technically and legally. ConvAD's ability to generate more transferable explanations across architectures and datasets gives it a practical edge over current state-of-the-art methods.
The paper benchmarks ConvAD against existing perturbation-based approaches across multiple architectures and datasets and claims consistent gains — though independent replication will be the real test, as self-reported SOTA comparisons in AI papers have a well-documented optimism problem.