Robots are getting better at generalizing what they grab, thanks to a new object representation called KeyGen.
Researchers built KeyGen, a system that learns canonical 3D keypoints from point-cloud scans of objects, then feeds those keypoints into a diffusion-based policy that predicts a robot's full manipulation trajectory. Instead of training a robot to memorize the exact shape of one mug or one box, KeyGen extracts consistent geometric landmarks that hold up across different instances of the same object category - different sizes, shapes, and poses. The team also built a photorealistic simulation benchmark with three manipulation tasks and a planning-driven pipeline to generate expert demonstration data across many object variants. In tests, KeyGen beat prior methods on both objects it had seen during training and ones it hadn't, and it held up when objects were rescaled, in both simulation and real-world trials.
This tackles a real bottleneck in robotics: behavior-cloning policies are notoriously brittle, often failing the moment an object's shape or pose drifts from training examples. If keypoint-based representations like this scale, it could cut down on the mountain of task-specific demonstration data manipulation robots currently need for every new object they encounter. That matters for anyone trying to deploy robots in unstructured environments like warehouses or homes, where no two objects look exactly alike.
Keypoint-based approaches to manipulation aren't new, but pairing them with a diffusion policy and testing across a dedicated category-level benchmark is a more rigorous validation than most academic robotics papers attempt. Still, the real-world results aren't described in much detail, so how well this holds up outside a lab bench remains an open question.