Researchers have built a more storage-efficient version of the tools used to reverse-engineer what's happening inside large language models.
Sparse autoencoders (SAEs) are a core technique in mechanistic interpretability — the field that tries to map neural network internals to human-readable concepts. The problem: as models grow, the dictionaries SAEs learn balloon in size, requiring huge numbers of stored values. The new design, called Expander SAEs, swaps out the standard dense decoder for one built on a mathematical structure called an expander graph, where each feature connects to only a small, fixed number of inputs. Tested on Pythia, Qwen2.5-3B, and Llama-3.2-1B, the most compressed version used 293 times fewer learned decoder values than a full dense decoder while recovering 84% of the baseline performance on a standard loss metric.
Interpretability research has quietly become one of the higher-stakes corners of AI development, with labs like Anthropic investing heavily in SAE-based tools to understand model behavior before deployment. The catch has always been scale: the same growth that makes frontier models capable also makes them expensive to study. A compression method that preserves most of the signal while cutting storage by two orders of magnitude could make serious interpretability work accessible at model sizes where it currently isn't practical.
The authors are careful to note the remaining gap isn't fully explained by parameter count alone — encoder design plays a role too, which means this isn't a free lunch, just a cheaper one.