AI/ ai · interpretability · machine-learning · research

AI Systems Already Track Which Parts of Them Mattered

A new arXiv paper shows adaptive-routing AI systems already log which components mattered, often more clearly at mid-level groups than at individual units.

Large AI systems that route work between components have been quietly keeping the receipts the whole time.

A new paper argues that adaptive routing systems, the kind used in mixture-of-experts language models, agent pipelines, and even telecom networks, already generate enough internal data to explain their own decisions. Every time such a system routes an input through a hierarchy of components, it multiplies together the routing weights chosen along the way. Multiply those weights down to a single leaf component and you get its attribution score. Stop partway up the hierarchy and you get the score for the whole branch above it, and the researchers show the two always add up consistently across LLM, Census, agentic, and telecom-network hierarchies.

That means teams no longer need a separate model just to explain which parts of a routing system did the work: the explanation is already sitting in the routing weights. The catch is what it actually measures. In the telecom test, the clearest picture came from Site- or Region-level readouts rather than individual Cells, and the researchers checked their numbers against Shapley values, the standard but computationally expensive method for testing counterfactual scenarios, to see where the two agree and where they don't.

The two rarely mean the same thing: this attribution shows what the router learned to prefer, not what a component is actually worth, so treat it as a map of habit rather than a verdict on merit.

TR

The Revision

Written by an AI system from the public sources credited above. How we write →