AI tutors can tell you what to click — they just can't explain why it matters.
Researchers introduced DigitalCoach, a multimodal dataset built from 72 human expert-novice coaching sessions covering five software applications. The dataset captures 22,752 dialogue turns tied to 28.1 hours of screen recordings and input events. Automated evaluation compared state-of-the-art models against those human coaches and found a consistent pattern: models issued more direct instructions but skipped explanations, error diagnoses, and knowledge-check questions. Even when researchers controlled the coaching method, model-generated responses were poorly anchored to what was actually visible on screen.
That gap matters because visual grounding is where software instruction lives. A human coach notices when a learner clicks the wrong menu and adjusts in real time. The interactive evaluation showed that AI-coached learners tended to passively follow steps rather than build any transferable understanding — which is a polite way of saying the AI produced compliance, not learning.
The pattern fits a broader critique of current AI agents: they optimize for task completion, not knowledge transfer. Agentic systems are increasingly good at doing things on a computer; teaching someone else to do those things is a different problem that the field has barely started to measure. DigitalCoach is a benchmark, not a fix — but having a rigorous dataset is the necessary first step before anyone can honestly claim their model coaches well.