Researchers have built a benchmark that catches AI computer agents behaving safely in theory but dangerously in practice.
The benchmark, called OSGuard, has two parts. The first grades individual proposed actions, labeling each as allowed, unrelated, or unsafe relative to the current screen state and the original user instruction. The second takes tasks from OSWorld, a standard agent evaluation suite, and seeds those environments with hidden hazards — files configured to be destructively overwritten if the agent takes a plausible-looking shortcut. The core finding: agents and guardrail models that scored well on the isolated action-judgment component consistently fell short when the same hazards were embedded inside a full, end-to-end task run.
Computer-use agents are now a shipping product at several major labs, and task completion rate has been the dominant scorecard. That metric says nothing about what got trampled on the way to the finish line. OSGuard's result — that safe individual-action judgment does not predict safe end-to-end behavior — is exactly the kind of gap that gets papered over when capability benchmarks are the only thing anyone is racing to top.
The researchers frame OSGuard as a diagnostic tool, not a verdict on any particular system. But the pattern it documents is familiar: the field learned with language model safety that benchmark-friendly behavior and real-world behavior diverge, and it tends to absorb that lesson only after the products are already shipping.