AI agents can stumble to the right answer for the wrong reasons — and a new research framework wants to catch that before it matters.
SkillCoach, described in a paper published this week, addresses a specific problem in how large language model agents are trained to use "skills" — reusable bundles that encode standard procedures, tool workflows, and domain rules. When a repository contains overlapping skills, agents frequently muddle through: selecting the wrong skill, skipping required steps, or composing workflows incorrectly, yet still passing the final check by luck or trial and error. SkillCoach replaces that blunt pass/fail signal with a four-dimensional rubric covering skill selection, skill following, skill composition, and what the authors call skill-grounded reflection. Crucially, it derives those rubrics from real agent rollouts rather than hand-authored rules, letting them evolve as the task distribution changes.
The distinction between process quality and accidental success matters more than it might sound. Most agent benchmarks reward outcomes, which means a model can score well while building habits that collapse on harder or novel tasks. By surfacing failures that final accuracy masks, SkillCoach offers a more honest training signal — and the researchers report it produces stronger supervision than outcome-only filtering.
This sits alongside a broader push to make agent evaluation less gameable; similar concerns drove earlier work on tool-use benchmarks that penalized shortcut paths. Whether evolved rubrics scale cleanly to messier real-world repositories remains the open question.