A new benchmark for catching malicious AI Agent Skills shows today's detectors mostly work only on threats they have already seen.
MaliciousSkillBench pools 13 public datasets of Agent Skills, the reusable instruction packages, scripts, and configs that extend LLM agents, then dedupes 8,414 raw malicious entries into 7,539 unique identities and lands on a final set of 9,740 Skills: 7,505 malicious and 2,235 benign across 11 harmonized attack categories. The researchers tested three learned text classifiers and three off-the-shelf Skill scanners. The best classifier, a word-level TF-IDF model paired with an SVM, hit a 93.2% Macro-F1 score on a standard random split, but that fell to 66.5% when evaluated on sources excluded from training, all while still flagging 62.4 percent of benign Skills as malicious. Off-the-shelf scanners fared no better, trading recall for lower false-positive rates instead of improving on both.
That gap matters because real attackers rarely publish through the same channels a detector was trained on. A tool that looks reliable on familiar data can still miss most new threats, or drown genuine alerts in false positives, the same failure mode that has dogged npm and browser-extension malware scanners for years. As Skill marketplaces expand, that source-disjoint blind spot is exactly where attackers will aim.
A scanner that only catches malware it has already met isn't a detector, it's a memory test.