A research team has taught robots to perform real-world manipulation tasks using nothing but a half hour of video of a human doing them by hand.
Researchers behind HumanEgo, an open-source framework described in a new arXiv paper, say the system converts ordinary egocentric video, footage shot from a person's point of view as they complete a task, into a representation robots can learn from directly. No robot hardware, teleoperation rigs, or robot-specific training data are required. With just 30 minutes of human video per task, the system hit a 92.5% average success rate across four real-world tasks; with only 15 minutes, it still managed 75%. The same policy transferred zero-shot to different robots, cameras, and environments without additional tuning.
Teaching robots by teleoperation, a human physically or remotely puppeting the robot through each motion, is slow and has to be redone for every new robot design. HumanEgo's pitch is that you can skip that step and reuse the same short video clips across hardware, which is the more useful comparison than the accuracy number alone. In the paper's own tests, HumanEgo beat matched-time robot teleoperation data by 41%, meaning the same 15-to-30 minute time budget spent recording a human, rather than driving a robot, produced a more capable policy.
Robot learning has spent years bottlenecked on the cost of collecting robot-specific demonstrations; if a phone-camera video of someone loading a dishwasher can substitute, that bottleneck gets a lot cheaper to work around. The code is open-source on GitHub, so outside labs can now test those numbers on their own hardware.