AI agents are getting bigger skill libraries, and picking the right tool from that pile turns out to be harder than it sounds.
A new paper describes SkillSight, a training-free retrieval framework built to help large language model agents choose the correct skill from a growing library of them. The researchers found that skill descriptions share so much boilerplate phrasing that standard retrievers get confused, scoring irrelevant skills as similar to a query just because the wording overlaps. SkillSight fixes this by identifying generic filler words through inverse document frequency, then downweighting those words in both the semantic embedding space and token-level lexical matching. On the SRA-Bench and SkillBench-Supp benchmarks, it improved Recall@10 by as much as 20.21 percentage points over a standard dense retriever, and ran up to 1,248 times faster than a dense-retriever-plus-reranker setup.
As agent frameworks add hundreds or thousands of callable skills, retrieval quality becomes the bottleneck, not the underlying model's reasoning. In end-to-end tests across three agent models, SkillSight also beat a simple LLM-based skill selector by up to 4.97 percentage points, with no extra training and none of the added latency of a reranker. That is the kind of unglamorous plumbing fix that could matter more for real-world agents than another benchmark-topping model release.
It's a reminder that agent frameworks are only as good as the index sitting between the model and its tools, and right now that index is mostly duct tape.