Medical AI can get the right answer at the wrong time — and that gap could cost lives.
Researchers have released MedStreamBench, a benchmark designed to test whether AI models can make time-sensitive decisions from medical video, not just score well on static question-answering. The benchmark pulls together 22 medical datasets and 5,419 QA instances across four temporal settings: retrospective, present, future, and proactive. Models are restricted to what they could actually see at a given moment in a video stream, rather than the full clip — the way a real clinical AI would have to operate. A proactive monitoring mode goes further, requiring models to decide whether and when to trigger alerts, not just answer when asked.
The results expose a real problem. Every leading general-purpose and medical vision-language model tested showed a sharp drop in performance when shifted from offline, full-video evaluation to streaming and proactive settings. That gap matters because nearly every existing medical AI benchmark measures what a model knows, not whether it can act on that knowledge at the right moment — a distinction that is irrelevant in a research paper but critical in an operating room or ICU.
Most AI safety conversations in medicine focus on accuracy. MedStreamBench reframes the question: accuracy at the wrong moment is still a failure. The benchmark is available on Hugging Face, which should lower the barrier for other labs to test their models before claiming clinical readiness — a bar the field has been setting embarrassingly low.