Multimodal AI models can now be interrogated on exactly which pairs of features across different inputs drove a given prediction.
Researchers released FL-I2MoE, a Mixture-of-Experts layer that slots into frozen pretrained encoders and sorts cross-modal evidence into three buckets: unique contributions from a single modality, synergistic pairs where two features combine for more than either offers alone, and redundant pairs that act as backups for each other. To test whether the identified pairs actually matter, the team built a masking pipeline that removes top-ranked pairs and measures the resulting performance drop. Across three benchmarks - MMIMDb, ENRICO, and MMHS150K - removing pairs flagged by their scoring methods hurt accuracy more than removing randomly chosen pairs with the same computational budget, suggesting the rankings reflect real causal structure.
Most explainability work on multimodal models stops at highlighting important tokens inside a single modality - it tells you the model noticed a dog in the image and the word "dog" in the caption, but not why that pairing mattered more than any other combination. FL-I2MoE's explicit separation of synergy from redundancy gives practitioners a vocabulary and a tool to audit these models at the interaction level, which becomes more pressing as multimodal systems take on higher-stakes decisions in medicine, content moderation, and legal review.
The code is public on GitHub, which is the right move for a technique that lives or dies by independent replication - interaction-level explainability is a crowded research space, and peer scrutiny will determine whether these scores hold up outside the authors' own benchmarks.