Researchers tested whether large language models could reliably write and interpret cloud access control policies — with mixed results.
The study evaluated multiple LLMs on two tasks: generating access control policies from written specifications, and summarizing what an existing policy actually permits. On the generation side, standard (non-reasoning) models produced syntactically valid policies but matched the intended specification only 45.8% of the time — meaning more than half their outputs were either too permissive or too restrictive. Reasoning-focused models fared significantly better, hitting 93.7% accuracy. The researchers also introduced a semantic summarization method that uses LLMs alongside symbolic analysis tools to characterize what requests a given policy allows, with more promising results.
The gap matters because access control policies are high-stakes: a policy that grants more access than intended is a security hole, and one that grants too little quietly breaks production systems. Cloud environments run thousands of these policies, and administrators write them by hand today — errors are common and hard to catch before something goes wrong. A reliable LLM-assisted tool could cut that risk, but 45.8% accuracy from off-the-shelf models is not a number you deploy in front of sensitive data.
The findings land at a moment when cloud providers and enterprise security teams are actively hunting for ways to automate policy management at scale. Reasoning models closing the gap to 93.7% is genuinely encouraging, but that remaining 6.3% error rate on security-critical infrastructure will require a much harder look before anyone ships this in production.