A new benchmark called S-EMBER reveals that AI models can't reliably pinpoint when something happened in a video stream, even as they get better at understanding what happened.
Researchers built S-EMBER around 3,141 videos totaling 388 hours of footage recorded through Ray-Ban Meta smart glasses. The dataset includes 9,448 question-and-answer pairs, each requiring a model to identify a specific moment in an ongoing video feed rather than scrub through a finished file. That distinction matters: most existing benchmarks hand a model the entire video upfront, which is nothing like how a wearable device actually works. S-EMBER forces models to operate causally, recalling only what has already streamed past.
The benchmark surfaces what the researchers call a localization paradox. Scaling up model size, resolution, or frame sampling rate improves semantic reasoning - the ability to describe or summarize events - but does nothing meaningful for temporal grounding, the ability to say "that happened at 2:14 PM, not 2:22 PM." That gap is architectural, not a matter of throwing more compute at the problem. For anyone building wearable AI assistants that need to answer questions like "where did I leave my keys this morning," this is the specific capability that remains broken.
The field has been racing to build always-on AI companions tied to smart glasses and similar hardware, with Meta, Google, and a handful of startups all placing bets. S-EMBER is a useful corrective: it sets a hardware-authentic baseline and makes clear that the retrieval problem is not close to solved.