A new benchmark suite shows that surgical AI models with identical accuracy scores can make mistakes of very different severity - and that matters when the mistake is misjudging what stage of an operation you're in.
Researchers built SurgHiBench, the first evaluation suite that judges surgical video-language models on the actual phase-to-step hierarchy of an operation, rather than just flat accuracy at one level. It tests three things: recognition, whether predictions stay consistent across granularity levels, and how severe an error is when a model gets it wrong. The team also built HyperSurg, a model that uses hyperbolic geometry and entailment cones to mathematically enforce that a surgical step sits inside its correct parent phase. They ran a general-purpose CLIP model, a standard Euclidean surgical model, and HyperSurg across four existing datasets covering three procedure types.
The headline finding: two models can post the same accuracy number and still fail in very different ways, from confusing two steps within the correct phase to guessing an entirely unrelated phase of the surgery. That distinction matters for any tool meant to monitor or assist in an operating room, where a near-miss error is a shrug and a wrong-phase error is a real safety concern. HyperSurg's hyperbolic geometry nudged predictions toward the correct procedural neighborhood, and the size of that improvement tracked how tree-like each dataset's phase-step hierarchy was - a useful signal for when the extra geometric complexity is worth it.
In other words, don't trust a leaderboard score to tell you whether an AI's mistakes are harmless or dangerous - ask what kind of wrong it is.