A new AI framework turns raw video of a robot handling objects into a ready-to-use physics simulation.
Researchers have introduced Agentic Real2Sim, a framework that uses vision-language agents to convert real-world recordings of robot-object interaction into simulatable digital twins, then hands those twins off for policy fine-tuning and evaluation. The pitch is that today's real2sim conversion is manual and brittle: it strings together separate visual-perception tools, mesh cleanup, and coordinate-frame alignment by hand. Agentic Real2Sim replaces that glue work with agent-driven automation and was tested across rigid-object manipulation, deformable-object interaction, and humanoid motion, three domains that normally need separate pipelines. The researchers also report that the framework's agentic decisions can run on an open-weight vision-language model at a small fraction of the cost of frontier models, while reaching a comparable conversion success rate.
The more interesting claim isn't the automation itself, it's the consolidation. Real2sim has been a bottleneck for robot learning partly because every new object type or motion domain has typically needed its own tooling. Folding rigid, deformable, and humanoid conversion into one agentic workflow, running on cheap open-weight models, lowers the cost of generating training and evaluation data for robot policies, not just building one-off simulations.
The underlying paper does not publish the actual success-rate numbers behind that cost comparison, so treat 'comparable to frontier models' as a self-reported result until outside groups put the released code, hosted on the project's site, through independent testing.