A research team built a benchmark and a model to teach AI agents to hear in 3D, not just see.
The researchers released OmniEchoBench, a benchmark of six tasks built from 197 real-world scenes that pair audio with video, plus 2,972 question-and-answer pairs testing whether a model can identify where a sound is coming from and what it means. A separate navigation set of 900 samples uses first-order ambisonics, a recording format that captures directional audio from all sides, gathered across 30 real environments. To generate enough training data, the team also built a rendering pipeline that keeps sound sources, camera views, and an agent's path geometrically consistent with each other. Their model, OmniEcho, pairs a dedicated spatial-audio encoder with a pretrained audio-understanding module, and it posted state-of-the-art results on the perception benchmark while nearly matching vision-only navigation systems when guided by sound alone.
Most embodied AI research treats sound as background noise or ignores it entirely, leaning on cameras and lidar to make sense of a room. A model that can point toward a sound without seeing its source could help a robot find a ringing phone in another room, react to something happening behind it, or work in the dark and in visual clutter where cameras struggle. That's a genuinely underused signal, and this paper is one of the first to give it a rigorous benchmark rather than an anecdote.
Still, close to vision-level navigation isn't better than it, and a 30-environment lab dataset is a long way from the acoustic chaos of an actual living room or warehouse floor.