A new study suggests that the way language models generate harmful content is more organized internally than anyone expected — and that organization may be the reason safety training keeps breaking.
Researchers used targeted weight pruning as a causal probe to map how harmfulness is encoded in large language models. They found that harmful content generation depends on a small, concentrated cluster of weights that is shared across different harm types and is separate from the weights responsible for ordinary, benign capabilities. Aligned models — those that have undergone safety training — showed a more compressed version of this harm cluster than unaligned ones, meaning alignment reshapes the internal structure even when it fails to hold at the surface level. The team also found that a model's ability to generate harmful content is dissociated from its ability to recognize or describe that content.
This matters because it offers a mechanistic explanation for two persistent headaches in AI safety: jailbreaks and emergent misalignment. If harmful capabilities are compressed into a tight weight cluster, fine-tuning on any narrow domain that touches those weights can unlock broad misbehavior — which is exactly the pattern researchers have observed. Pruning that specific cluster in one domain, the study shows, substantially reduced misalignment across others.
For years, AI labs have treated safety as a behavioral problem — add more RLHF, filter more outputs, patch the latest jailbreak. This research suggests the real target might be structural: a coherent internal representation of harm that exists whether or not surface guardrails are in place. That's a more tractable problem, but also a more unsettling one.