Vision-language models often 'know' a meme is hateful somewhere inside their weights - they just can't get that knowledge to their final answer.
Researchers tested Gemma-3 and Qwen3.5 using sparse autoencoders, role-conditioned probes, causal interventions, and recovery experiments across six harmful-content benchmarks, including Spanish and Hindi-English code-mixed cases. They found that reading the internal sparse features produced far better harmful-meme classification than the models' own native outputs: Qwen's sparse readout averaged 0.740 macro-F1 versus just 0.432 natively, and Gemma jumped from 0.532 to 0.714. A case study on Gemma-3-12B with Facebook's Hateful Memes dataset found a rank-32 image-prompt interaction pattern that scored 0.756 versus the model's native 0.685 macro-F1.
This points to a different bottleneck than the usual explanation for moderation failures. The models aren't necessarily missing the evidence that a meme is harmful - they fail to route that evidence to the output layer. The team showed calibration-only routing fixes recovered 93.3 percent of that accuracy gap without retraining the whole model.
Still, the fix isn't free. Distilling the probes into LoRA adapters helped on individual tasks but caused negative transfer when shared across tasks - patching one blind spot opened another.