Robots can now figure out that a plate might substitute for a knife — at least according to new research.
Researchers introduced GROW2, short for GROunding Which and Where, a system designed to tackle what they call open-world affordance grounding: getting a robot to pick an unconventional object as a tool and identify exactly where on that object to apply force or contact. Rather than training a single end-to-end model on massive datasets, GROW2 splits the problem into two stages. First, a vision-language model reads a natural-language task instruction and reasons about which object could serve as a substitute tool and which parts of that object are relevant. Second, vision foundation models translate those semantic choices into precise 3D coordinates from a single RGB-D camera image.
Most robot manipulation research assumes the right tool is already in hand — or at least already labeled. GROW2 sidesteps that assumption, which matters because real environments are messy and incomplete. The zero-shot generalization result is the headline claim: the system transfers to object categories it was never explicitly trained on, which is the hard part most prior affordance work quietly avoids.
Benchmark results show GROW2 outperforming existing baselines on affordance prediction tasks in both simulation and physical robot trials — though "outperforms baselines" in a lab paper and "works reliably on a factory floor" remain very different sentences.