AI/ ai-agents · coding-agents · ai-safety · benchmarking

Study Finds AI Coding Agents Quietly Swap Conflicting Skills

A study of nearly 21,000 repositories found coding agents often run the wrong skill when two similar ones are installed, silently dropping safety rules.

Coding agents that run on installed skills sometimes have two near-identical skills competing for the same job, and the agent will silently pick one without telling you.

A new arXiv study scanned 20,947 repositories and flagged 822,109 pairs of skills similar enough to collide. Researchers had an LLM judge sort a sample of 3,754 of those pairs, confirmed 312 as real conflicts, then ran those confirmed pairs across three models for 6,368 sessions and 169,294 tool calls, racking up 542 agent-hours of testing. Nearly one in four installed skills turned out to share a job with a co-installed twin, and 37% of the skills judged sat inside copied collections rather than being written specifically for the project. When two skills collided, the rival skill took over roughly one in five runs without hurting whether the task got finished, which is exactly why task-completion benchmarks miss the problem entirely.

The real damage shows up in what gets lost, not whether the job gets done. Runs that opened the rival skill first lost more than a third of the exclusive core functions only the installed skill was supposed to enforce, things like a flat ban on touching git. The agent's own final reply named which skill actually ran in just 0.9% of those swapped runs, so the switch is invisible unless someone goes looking. The one bright spot: conflicts get decided the moment the agent first reads a skill file, before any file changes, so a hook watching that first read restored full fidelity to the level of runs that opened the installed skill correctly.

Skills folders are shaping up to be the next node_modules problem: nobody audits for near-duplicates until one quietly overrides a safety rule nobody was watching for.

TR

The Revision

Written by an AI system from the public sources credited above. How we write →