A new AI framework called Exo2EgoHOI turns third-person videos of people handling objects into first-person footage, while keeping track of exactly how hands grip and move those objects.
Researchers built Exo2EgoHOI to translate exocentric video (shot from the side, or by someone watching) into egocentric video (as if seen through the actor's own eyes). The system combines scene geometry, rendered hand poses, and a dense map of hand-object relationships into what the team calls a 4D hand-object interaction prior, then injects those cues into the video generator through a dual-branch adapter. A separate attention mechanism, which the researchers call Decomposed Gated Cross-Attention, keeps the object itself visually consistent even as the camera viewpoint shifts dramatically. On the ARCTIC-HOI benchmark, the method improved object-tracking accuracy (mIoU) by 32.3% and cut two measures of hand-pose error by 34.7% and 50.0% compared to the best prior approach.
The gap this targets is real: training robots and other embodied AI systems to manipulate objects works best with first-person video, but most existing footage of humans using their hands was filmed from a third-person angle. Earlier viewpoint-conversion tools could make a clip look like it came from a different camera position, but they tended to lose track of exactly how a hand gripped or released an object - the one detail a robot actually needs to learn from. Preserving that contact detail across a full viewpoint change, rather than just the visual style, is the specific problem this paper claims to solve.
The gains, though, are measured on two curated lab benchmarks built for controlled manipulation tasks, not the cluttered, unpredictable footage robotics teams actually want to mine from the open web - turning a lab result into something that scales is the harder part of this problem.