Robots powered by vision-language-action models fail more often than their demos suggest, and a new detection framework aims to catch those failures while they are happening, not after.
Researchers behind a system called Hide-and-Seek trained a failure detector using only trajectory-level labels - a single pass or fail tag for an entire robot run - instead of expensive per-step annotations or resampling every action. The framework applies contrastive learning both across and within trajectories to pinpoint which specific actions in a failed run actually caused the problem, turning one coarse label into a localized failure signal. The team tested it on three VLA policies, OpenVLA, pi0, and pi0.5, across two simulation benchmarks, LIBERO and VLABench, plus a real robotic platform, and reported state-of-the-art multi-task failure detection results, including generalization to tasks the model had not seen during training.
VLA models let robots follow plain-language commands, but they are notoriously unreliable in execution, and existing failure detectors either need extra model calls to resample actions or apply one blunt label to an entire trajectory, obscuring exactly when things went wrong. Hide-and-Seek uses conformal prediction to explicitly balance accuracy against timeliness rather than claiming a win on both fronts - a real-world robot that flags an error three steps too late has already dropped what it was carrying.
That trade-off framing is unusually honest for a benchmark paper: the claim is not that it catches everything, but that it catches more, sooner, with less supervision than prior approaches - a narrower win, but a more useful one for anyone running robots outside a lab.