Researchers have published the first open benchmark designed to test whether AI agents can execute a full enterprise network intrusion, not just clear a capture-the-flag room.
AgentCyberRange packages 110 vulnerabilities across 15 real web applications and 8 enterprise-like networks containing 156 internal hosts. The benchmark runs AI agents through two sequential phases: web exploitation, where the model must discover and validate vulnerabilities in exposed applications, then post-exploitation, where it converts that initial access into broader control over internal systems. Six frontier AI systems were evaluated under matched prompts and identical token budgets. GPT-5.5 paired with Codex led the field, completing 16.1% of web exploitation tasks and 31.7% of post-exploitation tasks; with concrete hints supplied, those figures climbed to 33.0% and 46.3%.
The gap this fills is real. Prior public benchmarks tested isolated capabilities — CTF challenges, single-host vulnerability reproduction, standalone exploit generation — but abstracted away the multi-step intrusion chain that makes attacks actually dangerous. Making that chain reproducible and open is what lets defenders track capability growth before it surfaces somewhere worse. The more alarming detail is buried in the results: during testing, models uncovered previously unknown vulnerabilities in widely used open-source projects and generated payloads that bypassed host-level defenses — findings the benchmark was not designed to elicit.
Success rates in the teens might sound reassuring. Benchmarks like this tend to age faster than the reassurance does.