A small AI model trained on a new synthetic dataset nearly doubled its benchmark score on computer-use tasks and beat GPT-4o along the way.
Researchers introduced ISE, a three-stage pipeline for generating OS-agent training data. The first stage constructs roughly 44,000 unique task intents by crossing user personas with domains, task types, and complexity levels. A role-locked simulator then runs multi-turn conversations between a synthetic user and an agent, where each user turn responds to real execution outcomes rather than scripted placeholders. In the final stage, every tool call runs inside a live, isolated OS workspace, producing authentic failure-recovery sequences. The resulting dataset, called ISETrace, contains 23,132 trajectories averaging 68 dialogue turns each.
Fine-tuning Qwen3-8B on ISETrace pushed its score on the ClawEval agent benchmark from 19.3 to 37.7. That fine-tuned 8-billion-parameter model outperformed GPT-4o running zero-shot and Qwen3-32B, a base model four times its size. The implication is direct: for OS agents, data quality matters more than parameter count, which cuts against the assumption that bigger models are the default answer for computer-use tasks.
The code and dataset are publicly released, notable in a field where training data is often the most jealously guarded asset. Whether ClawEval gains translate to the messier reality of a real desktop, with its conflicting apps, permission dialogs, and half-finished tasks, is the question the benchmark cannot answer.