Researchers have released SynthSAEBench, a benchmark toolkit designed to close a long-standing gap in how Sparse Autoencoders are evaluated.
Sparse Autoencoders, or SAEs, are a tool interpretability researchers use to decompose what a neural network has learned into human-readable features. The problem is that existing benchmarks for judging SAE quality are caught between two bad options: tests run directly on large language models produce too much noise to tell good architectures from slightly better ones, while small synthetic experiments are too toy-like to mean anything. SynthSAEBench splits the difference by generating large-scale synthetic data that mimics realistic feature properties — correlation, hierarchy, and superposition — while also supplying ground-truth labels so researchers can check whether an SAE actually learned the right thing, not just something plausible-looking.
That ground-truth access matters more than it might sound. The benchmark confirms several phenomena already observed on real LLMs — including a disconnect between how well an SAE reconstructs inputs and how good its learned features actually are — which means findings here are likely to transfer. It also catches something new: a class of SAEs using a technique called Matching Pursuit can game reconstruction scores by exploiting noise in superposition, learning nothing genuinely useful about the underlying features. That kind of overfitting would be invisible to benchmarks without ground truth.
SAE research sits at the heart of mechanistic interpretability, the field trying to make large models legible before deploying them at scale. Better benchmarks are infrastructure work — unglamorous but load-bearing. The fact that a plausible-sounding architecture can score well while learning the wrong features entirely is exactly the kind of silent failure that keeps interpretability researchers up at night.