A research team has built a violence-detection model that tells its audio encoder when not to trust itself.
Most audiovisual AI systems extract visual and audio features separately, then blend them at the end. AViS-Mamba flips that order. At every layer of the audio encoder, a compact visual representation injects a modulation vector that reshapes the encoder's internal temporal operators in real time. A routing gate then controls how hard that visual signal pushes. The result: when audio is noisy, dubbed, or missing entirely, the model leans on vision; when audio is clean and informative, it uses it. On two benchmarks - NTU-CCTV and DVD - the model hit 88.59% and 75.74% accuracy, claiming state-of-the-art results on both.
The practical payoff is resilience. Surveillance footage is exactly the environment where audio fails most often - wind, crowds, distance, and cheap microphones all degrade the signal. A system that degrades gracefully under those conditions is more useful than one optimized only for clean lab recordings. The paper's layer-wise analysis also shows the model doesn't apply one global policy; it adjusts its audio reliance differently at different depths, which suggests the architecture is doing something structurally interesting rather than just learning a simple on-off switch.
The researchers also introduce Adaptive AV-InfoNCE, a contrastive loss that learns the relative weight between audio-to-video and video-to-audio alignment rather than treating both directions equally - a small but telling sign that fixed assumptions about modality balance keep causing headaches in multimodal research.