A new study on AI-driven security tool orchestration found that the client software running the model matters more than the model itself.
Researchers used HexStrike-AI, an open-source orchestrator exposing more than 150 tools, as their testbed. They ran 774 trials across 86 picoCTF challenges, testing three tool-access regimes and three model-and-client configurations, then diagnosed failures and applied targeted fixes. The overall solve rate rose from 55.4% to 72.0%, with every configuration improving significantly. The biggest lever turned out to be the client: swapping the software driving a fixed DeepSeek model produced a 2.1x performance gap between two clients running the same underlying model.
Most AI security coverage fixates on the underlying model: which lab built it, how recent the training cutoff. This study shifts the frame. The orchestration client, the layer that decides what tools to call and in what order, may matter more than the model it drives.
The researchers note that the fixes were tuned on the same challenges they were evaluated on, and the client effect is demonstrated for one model only, so its generality, in their own words, remains a hypothesis.