A research framework called 3D HAMSTER shows that giving robot planners a sense of depth makes them meaningfully better at manipulation tasks.
Current hierarchical robot models split the job in two: a high-level planner figures out where to move, and a low-level policy executes the motion. Recent systems use 2D end-effector trajectories from a Vision-Language Model as the handoff signal. The problem is that the low-level policies actually operate in 3D metric space on point clouds — so when a 2D path arrives without depth information, the system guesses depth by reading whatever surface sits beneath each waypoint. That guess is often wrong, and the resulting trajectories are geometrically distorted. 3D HAMSTER fixes this by adding a depth encoder and a dense depth reconstruction objective to the planner, letting it output full 3D waypoint sequences that feed directly into the point-cloud policy.
The gap between 2D planning and 3D execution is a quiet but persistent source of robot failure, especially when lighting changes or an object appears in an unexpected position. By closing that gap at the architecture level rather than papering over it with post-hoc depth estimates, 3D HAMSTER posts its largest gains precisely in the hard cases: appearance-altering shifts and language or visual conditions the system has never seen before.
Robotics benchmarks are famously easy to game in simulation, so the real test is whether the gains hold on physical hardware — the paper claims they do, though independent replication will matter more than the project page.