A new benchmark stops grading AI agents on the final screen and starts grading every click along the way.
Researchers built OSWorld-Pro, an evaluation set of more than 300 computer-use tasks broken into over 2,800 smaller subgoals, checked against more than 67,000 human annotations. Unlike its predecessor OSWorld, which only checks whether an agent's finished output passes a functional test, OSWorld-Pro tracks whether the agent hit each intermediate step correctly, using LLM-judges tuned to match human raters. That process view exposes where agents actually go wrong, rather than just whether the end result happened to work. The team ran current models through it, including Claude Opus 5, one of the top-scoring models tested.
That gap matters because it exposes slack in how computer-use agent performance gets reported. A model can rack up wrong clicks and off-task detours during a job and still get credited with a win if the final file looks right. OSWorld-Pro's subgoal-by-subgoal grading catches exactly those detours, flagging failure modes like actions irrelevant to the current subgoal and imprecise clicking that a pass-fail final check would miss entirely. Claude Opus 5 scored 75.7% on OSWorld-Pro's subgoals versus 83.4% on the original OSWorld benchmark, a reminder that fewer errors sneak through when you check the whole path instead of just the destination.
It's the same lesson software testing learned long ago: a passing final check doesn't tell you whether the process behind it was actually sound.