AI safety guardrails are far worse at following complex, app-specific rules than the industry tends to admit.
Researchers introduced SafePyramid, a benchmark of 1,000 multi-turn conversations across 10 domains, paired with 3,000 application-specific safety policies containing 61,699 distinct natural-language rules. The benchmark tests guardrails at three difficulty levels: understanding individual rules, reasoning over rule dependencies, and adapting to entirely new policy frameworks defined in context. Ten frontier large language models and five policy-configurable guardrails were put through it. The best performer, GPT-5.5, correctly identified the full violated rule set in 54% of the easiest cases, 35% of mid-tier cases, and just 13% of the hardest.
This matters because real deployments do not use tidy predefined risk taxonomies. Enterprises write their own policies — often dense, layered, and full of interdependencies — and expect guardrails to enforce them reliably. A guardrail that misses 87% of complex policy violations is not a guardrail; it is a liability disclaimer with extra steps.
The findings land at a moment when AI providers are marketing configurable safety layers as enterprise-ready infrastructure. SafePyramid suggests the gap between the sales pitch and the actual capability is still wide enough to drive a truck through.