A 15-percentage-point jump in robot task success is large enough for users to actually notice, a new study finds.
Researchers compared two configurations of a multimodal robot-grasping system in a within-subject study with 24 participants, all performing the same tabletop object-grasping task. The baseline combined Whisper for speech recognition, Florence-2 for open-vocabulary object detection, and LLaMA 3.1 for language understanding, completing the task correctly 75% of the time. The improved version replaced perception and language modules with Grounding DINO paired with SAM and a Qwen-series model — the paper cites a specific version string that cannot be verified against any confirmed public Qwen release, so it is omitted here — reaching 90% end-to-end success. After using both systems, 17 of 24 participants (70.83%) preferred the improved configuration, a result significant at p = 0.043, and rated it higher on perceived speed, reliability, and overall competence, with large to very large effect sizes after Holm correction.
Robotics papers rarely check whether benchmark gains register with real people; this one did, and the answer was yes. That matters because it validates a methodological assumption the field mostly skips — that measurable improvements eventually cross a perceptual threshold users can articulate without being told which system is newer.
Twenty-four participants on one task is a data point, not a design rule. Whether the perception gap holds for smaller gains, or for tasks where the robot fails visibly, remains an open question.