AI agents still can't reliably work a microwave panel or press the right elevator button.
Researchers have released SWITCH, a benchmark designed to test how well AI agents interact with tangible control interfaces - think appliance panels, remote controls, and embedded touchscreens - across extended, multi-step tasks. The dataset spans 1,170 egocentric videos covering a range of real-world functional settings, annotated for instructions, actions, state changes, outcomes, and recovery behaviors. Unlike most existing benchmarks, which test perception or single-step responses in isolation, SWITCH evaluates closed-loop reasoning: the agent must act, observe what changed, and correct course if something went wrong. Both proprietary and open-source multimodal models were put through the benchmark, and both struggled.
The gap this exposes matters because embodied AI research has largely optimized for tasks that look impressive in demos - object recognition, single-command navigation - while ignoring the boring, friction-filled reality of using human-built environments. Pressing a button is trivial; pressing the right button, confirming the oven actually changed temperature, and recovering when it didn't is a different problem entirely. SWITCH is the first structured attempt to measure that full cycle at scale.
Most AI benchmark announcements are quietly optimistic about what the results imply for deployment readiness. This one is not - the paper describes the failures as "persistent," which is a measured way of saying the field has more work to do before robots can handle a hotel thermostat unsupervised.