A new benchmark tests whether AI agents can do the unglamorous work of figuring out what a neural network actually learned.
Researchers built SAEScientist-Bench to see if agents can use sparse autoencoders, a tool that isolates interpretable features inside a model, to hunt down a target concept in Gemma-2-9B-IT's Gemma Scope dictionary of over 131,000 features. Given a concept, an agent designs contrastive text probes, searches the dictionary, and picks its best-guess feature. That pick gets scored against expert reference features from Neuronpedia on three axes: activation rank, concept selectivity, and causal steering. Across 10 agent setups and 20 tasks, frontier agents showed genuine discovery ability and even led on some metrics, but landed well behind the expert baseline overall, especially at steering. The researchers also found agents frequently misread their own experimental results, even when their contrastive probes were sound.
This matters because the push toward recursive self-improvement has mostly automated training, not the auditing that would make autonomous development trustworthy. If agents are going to build and modify models with less human oversight, someone needs to check what those models actually learned - and this benchmark is a rare attempt to measure whether agents can do that checking themselves, using a tool (SAEs) that has become a standard part of interpretability research.
The agents were decent at ruling concepts out and worse at proving what a feature actually does - which is exactly the gap between spotting a correlation and confirming a mechanism. That gap is still where the humans win.