Robots that nail a demo in one room often flounder the moment the furniture moves, and a new benchmark from Nvidia researchers finally puts numbers on why.
The tool is called RoboLab, a simulation framework built to stress-test "task-generalist" robot policies - the foundation models meant to handle many tasks, not just one. It ships with RoboLab-120, a set of 120 tasks split across three skill categories (visual, procedural, relational) and three difficulty tiers, with scenes generated by humans or LLMs so they aren't tied to any specific robot or policy. The researchers then ran real-world policies through the simulator and measured two things: raw success rates, and how much performance degraded under controlled perturbations to the scene. The results showed a significant gap between how these models perform and how well they actually generalize.
That distinction matters because most existing robotics benchmarks let training and evaluation scenarios overlap, which inflates success rates and hides exactly this kind of fragility. RoboLab is explicitly designed to close that loophole, which is why its findings are less flattering than the leaderboard numbers robotics labs usually publish.
It's the robotics equivalent of a language model acing a benchmark it was quietly trained on. The industry has spent two years hyping "general-purpose" robot brains; this benchmark suggests the generalizing part is still mostly aspirational.