A new study says the search for "hallucination neurons" inside large language models is shakier than the original claims suggested.
Researchers built a five-step diagnostic protocol to stress-test claims that specific neurons can be isolated to flag when a model is about to hallucinate. They reran prior work on so-called H-neurons using two open-source models, Gemma 3 4B and MedGemma 4B, across three question-answering datasets: TriviaQA, BioASQ, and NQ-Open. The detection signal held up, and on TriviaQA and BioASQ it beat the original reported AUROC gaps, with Gemma 3 4B outperforming MedGemma 4B across the board (0.311 vs. 0.235 on TriviaQA, 0.474 vs. 0.455 on BioASQ, 0.128 vs. 0.112 on NQ-Open). Causal tests on 500 examples across five random seeds confirmed the effect was real rather than statistical noise. But the diagnostics also showed the neurons weren't cleanly isolated: 19 of 22 selected H-neurons in the Gemma 3 4B setups correlated strongly with other features, bootstrap resampling produced only moderately stable picks, and sparse versus dense neuron rankings barely agreed.
That gap, between detecting a signal and pinning it to one neuron, matters because interpretability findings like this get cited to justify auditing and steering model behavior. If the neurons behind a hallucination score are tangled up with dozens of other features, a fix that nudges one neuron can quietly nudge unrelated behavior too.
The paper's proposed remedy is unglamorous: run the same five checks before trusting the next paper that claims to have found the neuron for anything.