First-person activity recognition systems that combine camera footage with motion sensors are quietly ignoring half their data. A new research paper explains why and offers a fix.
These systems are designed to handle open-world conditions, meaning they need to flag activities they were never trained on while continuing to learn from new ones. Researchers found that when systems fuse camera (RGB) and inertial measurement unit (IMU) signals - the accelerometers and gyroscopes common in phones and AR headsets - the visual stream dominates the combined output, leaving IMU data effectively sidelined. That imbalance compounds over time: as a system absorbs new tasks continuously, a well-documented failure mode called catastrophic forgetting degrades earlier representations, and the already-secondary motion signal suffers most. The proposed framework, MAND, attacks this on two fronts - a component that adjusts each modality's contribution dynamically at inference based on per-sample reliability, and a training-time component that preserves each modality's discriminative capacity through modality-specific heads and logit distillation.
The open-world problem - what a deployed system does when it encounters behavior it was never trained on - is the unglamorous test that most academic benchmarks skip. Getting it wrong in a wearable context means a system either flags everything as novel (useless) or flags nothing (dangerous). The paper reports a substantially lower false-positive rate at a 95 percent true positive threshold, suggesting the "I don't know" signal becomes more trustworthy.
The code is public on GitHub, which is useful. The evaluation relies on a single public multimodal benchmark, so how these gains hold up across real-world mixes of environments - office, street, gym - is still an open question.