Getting AI models to use the right tool, at the right time, is harder than it sounds.
Researchers have published a paper introducing PORTS, a training method for the retrieval systems that pre-select tools before a large language model ever sees them. The core problem: LLMs can theoretically call hundreds of external tools, but stuffing all their documentation into a prompt is slow and expensive. Existing retrievers help narrow the list, but they are trained independently from the LLMs they serve, which means what the retriever considers relevant and what the model actually needs often diverge. PORTS addresses that gap by using a signal derived from the LLM's own perplexity — essentially how surprised the model is by a given tool — to fine-tune the retriever so its rankings correlate with real downstream task performance. A contrastive loss on documentation strings is applied at the same time to keep similar-sounding tools from getting conflated.
The alignment problem PORTS targets is underappreciated outside research circles. As agentic AI systems grow more common, the tool catalogs they draw from are expanding fast — think enterprise deployments where a single assistant might have access to dozens of APIs. A retriever that was trained on generic text similarity will consistently surface plausible-sounding but wrong tools, silently degrading performance in ways that are hard to diagnose. PORTS' approach of borrowing a preference signal directly from the frozen LLM is a practical shortcut that avoids retraining the entire stack.
The method was tested across six datasets, two encoder architectures, and three LLMs, with the authors reporting accuracy gains at low computational cost. That last claim deserves scrutiny — "low computational cost" is a phrase that tends to look different in a lab than in production — but the cross-model generalization results at least suggest PORTS is not overfit to one narrow setup.