A new paper argues that multimodal sentence embeddings are not just outputs — they are witnesses to their own errors.
The researchers focused on SONAR, a model that produces sentence-level embeddings across languages and modalities. They found that certain dimensions within those embeddings respond measurably to perturbations during the encoding-decoding cycle. By checking whether what went in matches what comes back out, the team built a detector that flags decoding anomalies without needing a separate monitoring system. They also experimented with nudging those sensitive dimensions to try to correct errors in place, not just catch them.
This matters because multimodal models are increasingly deployed in pipelines where silent failures — a garbled translation, a misrepresented audio segment — are hard to catch downstream. An anomaly detector baked into the embedding space itself is cheaper than bolting on an external watchdog, and it works closer to the source of the problem. Most reliability work in this space focuses on outputs; this paper looks one layer earlier.
The approach is narrow — it targets SONAR specifically and sentence-level embeddings in particular — so whether the findings generalize to other multimodal architectures is still an open question.