A new evaluation method roughly doubles how many real-world failures researchers can find in autonomous systems, without burning through expensive real-world test runs.
Testing self-driving cars, robotic arms, and legged robots for every failure mode is slow and expensive, so researchers often lean on cheap proxies: simulators, lower-fidelity systems, or related policies that can be sampled thousands of times. The problem is that failures found in those proxies frequently do not transfer to the real target system, because of the well-known sim-to-real gap and quirks specific to each system. A new research paper corrects for that gap by learning a local predictor of real-system risk that adjusts proxy failure signals using control-variate-inspired residual modeling, then picks which scenarios to test next with a support-aware mutual-information objective favoring realistic, well-supported failures while still covering different failure types. Across driving, manipulation, and quadruped velocity-tracking tasks, it found up to twice as many failures as random sampling and standard active-learning baselines, including severe and varied failures the other methods missed.
Failure discovery, not average performance, is usually what gates whether an autonomous system is safe enough to deploy, and real-world testing budgets are small precisely because failures are rare and expensive to reproduce. A method that squeezes more, and more diverse, failures out of a fixed budget of real-world tests is effectively free reliability information.
Papers reliably beat their own baselines; the real test is whether a self-driving or robotics safety team finds this more useful than simply running more simulations.