A research team built a system that watches people use software, then turns those recordings into skills an AI agent can reuse.
The framework, called Work2Skill, includes a tool named WorkRecorder that captures screen video, mouse and keyboard events, and accessibility-tree data while a human performs a task. From that, the researchers built WorkDemo-350, a set of 350 screened desktop demonstrations covering real GUI workflows. An agentic method then reviews those recordings and extracts reusable skills - workflow rules, step order, and checks for whether a task actually finished - instead of just replaying clicks. Four different AI models given these skills scored higher than the same models given no skills at all.
Computer-use agents stumble on unfamiliar software because they lack the tacit knowledge a human builds up through experience. Converting that know-how into transferable skill files, rather than hand-coding automation scripts for each app, is a more general fix. The reported gains - 4.67 to 10.54 points on one benchmark and 3.11 to 7.50 on another - are modest but measured against a clear baseline, and they come without retraining the underlying model.
It is an incremental step, not a solution to the deeper problem: these agents still need a human to demonstrate the task correctly at least once before they can learn anything from it.