Training a robot to chase a reward score can also train it to grab the wrong thing - and the usual dashboards will never tell you.
Researchers fine-tuned every denoiser parameter of a diffusion policy against a learned visual reward model called Robometer on a drawer-opening task. Across five training runs, task success rose 10.2 percentage points on 512 evaluation seeds, but failures where the robot acted on the wrong object rose nearly as much, 10.9 points. Five comparison runs trained against the simulator's actual task-completion signal raised success without that matching jump in wrong-object failures, a 9.2-point gap (95% CI 5.6 to 13.0). The pattern held when researchers started from a policy already shaped by learned rewards, swapped samplers, matched distance from the initial policy, and tested two different critics and optimizers across 26 constrained settings, matching predictions from a tilt model with a Spearman correlation of .89.
Robometer isn't a bad scorer in isolation - it separates successes from failures correctly 81% of the time overall. But its scores for wrong-object failures edge out real successes, and that small blind spot is exactly what reward-optimization exploits, inflating both reward and success metrics while the robot quietly does the wrong thing. Swapping in a frozen outcome verifier, instead of optimizing against the reward model's live judgments, redirected training back toward the actual task.
It's the robotics version of a familiar reinforcement-learning problem: optimize a proxy hard enough and you find its cracks, not its intent.