AI interpretability research has a blind spot: the models it studies are better at hiding their workings than the tools used to find them.
Researchers studying how transformer models produce behavior typically remove one component at a time and measure the effect. The problem is that transformers can self-repair — when a key circuit is knocked out, a dormant backup silently takes over, making the original component look unimportant and the backup look irrelevant. Both conclusions are wrong. A new paper introduces Conditional Co-Ablation (CoAx), a scoring method that removes a primary component first, then measures how much each remaining component's behavior changes as a result. That second-order signal is what exposes the backups that single-unit testing misses.
The stakes are higher than academic tidiness. Interpretability tools are used to guide model pruning, capability removal, and safety attribution — the processes researchers use to locate and disable specific behaviors in AI systems. If those tools systematically miss backup circuits, pruning a capability may not actually eliminate it, and safety attributions may point at the wrong components entirely. On GPT-2-small, CoAx raised backup-head recovery from an ROC-AUC of 0.33 to 0.91, outperforming gradient-based self-repair-aware scores that topped out at 0.82, and the method transferred across eight models.
The finding lands at a moment when mechanistic interpretability is being leaned on heavily as a foundation for AI safety work — which makes it uncomfortable that a core assumption of the field, that component importance is additive, turns out to be wrong in exactly the cases where robustness matters most.