Researchers have found the exact spot inside a vision-language model where it decides that a photo and a caption together cross into unsafe territory.
The team built SSU-Bench, a dataset of matched safe and unsafe image-text pairs created by swapping a single word in the prompt or editing one annotated region of an image. Testing three vision-language models, they transferred internal activations between paired examples and tracked how the model's safety verdict shifted. Edits at the specific input position that changed moved the needle early in the decoder's layers, while the model's final summary token only became influential later. A simple linear readout of that final-token state could predict the model's own verdict - right or wrong - and the pattern held up across all three models tested.
That last part is the unsettling detail. The readout predicted incorrect safety judgments just as reliably as correct ones. In other words, the model's internal "decision" is legible to researchers well before anyone checks whether that decision is actually right. For teams building content moderation or child-safety filters on top of these models, that gap between a readable verdict and a correct one is exactly where false negatives slip through unnoticed.
Interpretability work like this tends to get filed under academic curiosity, but it is really an audit tool. Knowing when a model locks in its answer - and that the answer is predictable from a single vector - gives developers a concrete point to intervene, rather than treating safety filtering as a black box patched with more training data.