A new robot-training method lets human demonstrators watch a live simulation of a robot hand's grip while they move their own hand.
Researchers built Touch2Robot, described in a paper posted to arXiv. It captures human hand motion, tactile-glove data, and object motion during manipulation, then trains object-specific reinforcement-learning policies that reproduce the demonstrated motion using contacts the target robot hand can actually make. A real-time retargeter converts the human's live movements into a simulated robot-hand pose, synced with the tracked object, so the demonstrator sees where the robot would touch and can adjust their grip accordingly. Across four real-world tasks, this feedback loop raised real-robot replay success from 37.9% to 72.1%, and cut the time needed to collect each usable demonstration from 58.6 seconds to 18.2 seconds.
Robot-hand data collection has long forced a choice between cheap human demonstrations that do not transfer well to hardware, and expensive robot-only demonstrations that transfer but do not scale. Showing the human a live preview of the robot's simulated contacts closes that gap without new hardware costs, and downstream policies trained on this data improved 29.1 percentage points over a visual-only baseline.
The simulated contacts matched real robot tactile readings only 44.2% of the time, a reminder that a preview is still a preview, not the real thing.