A new paper argues that a popular technique for steering AI behavior has a much narrower useful range than researchers assumed.
Researchers tested sparse autoencoder (SAE) feature ablation — a method that targets specific learned "features" inside a model to suppress harmful outputs — on Google's Gemma-2-9B-it model. They found that intervening on a small set of features (around 800) can reduce harmful compliance with relatively low collateral damage. Push that number to 1,600 and the method loses ground to simpler dense baselines. Push it to 3,200 and the model stops producing coherent text altogether. The team also flagged a measurement problem: automated safety judges were counting incoherent gibberish as "unsafe," inflating apparent success rates. Their fix — only counting outputs that are both judge-unsafe and coherent — substantially changes what counts as a win.
SAE-based interventions have attracted serious interest as a mechanistic, interpretable alternative to blunt refusal fine-tuning. If specific features map reliably to harmful behavior, the thinking goes, you can surgically disable them at runtime. This paper complicates that picture: the "surgical" regime is narrow, and the features driving it are concentrated at the top of a rapidly decaying activation curve.
The upshot is a familiar one in AI safety research — a technique that looks clean in limited trials gets messier under stress. SAEs may still be useful, but as regime-dependent tools rather than general-purpose safety handles.