A research team built a robot world model that learns manipulation from synthetic data, then transfers that knowledge to real hardware using a fraction of the real demonstrations normally required.
Mask2Real-WM splits the prediction problem into two stages. A dynamics model takes past segmentation masks and a sequence of 23-degree-of-freedom joint actions, then predicts what those masks will look like in the future. A rendering model, built on Stable Video Diffusion with a ControlNet backbone, converts those predicted masks into photorealistic images. The key ratio: the dynamics model was pretrained on over 50 hours of synthetic simulation footage, then fine-tuned on fewer than 2.5 hours of real robot demonstrations.
Sim-to-real transfer has been robotics' stubborn unsolved problem for years. Raw-pixel world models absorb textures, lighting, and sensor noise that differ sharply between simulation and reality. Segmentation masks strip all of that out, leaving only shape and motion — a space where synthetic and real data are similar enough that cheap simulation pretraining can do genuine work.
This is a benchmark result on a pick-and-place task, not a shipping product. But a roughly 20-to-1 ratio of synthetic to real training data is a number robotics labs will study closely as they try to cut the cost of real-world data collection.