Researchers tested whether monitoring AI models' internal activation signals can reliably flag harmful requests — and the answer is "sort of."
A team probed the residual stream — the layer-by-layer activation states — of several 7-8 billion parameter language models to see if those signals could separate harmful prompts from benign ones that look similar on the surface. The sensors blocked 95.5 to 97.7 percent of judge-classified attacks. But the same system also blocked 59.6 to 68.4 percent of XSTest prompts — a benchmark of benign requests engineered to resemble dangerous ones — meaning the detectors fired on a lot of innocent traffic. When researchers tuned classifiers on matched harmful-benign pairs, false-block rates climbed to between 79.6 and 100 percent of legitimate requests in some test conditions.
The core problem is context: a request can be harmful or harmless depending on who's asking and why, and activation probes can't reliably make that call. Any safety filter that can't read context will either let too much through or block too much — and at scale, both failures are costly.
The paper lands on an honest label for these tools: "broad-risk detectors," not context adjudicators. That's a useful boundary to draw before anyone markets one of these probes as a complete content-safety solution.