A new audit framework scores AI systems for security risk using fixed, repeatable rules instead of a reviewer's gut call.
Researchers built a tool that reads engineering artifacts, things like CI pipeline configs, logging setups, and access controls, and maps them onto MITRE's ATLAS catalog, the AI-focused counterpart to the ATT&CK framework for attack techniques. Each project gets scored on a four-level scale for every technique in ATLAS, with every score tied back to the specific evidence that produced it. The scoring logic itself is formally verified. The authors mathematically proved the evaluator can't skip cases, contradict itself, or behave inconsistently as more controls get added. They tested it on five open-source AI projects, then re-ran the scores after bolting on a software bill of materials generator and CI security scanning gates to see if the numbers actually moved.
AI security reviews today mostly run on checklists and whoever is holding the clipboard, so two audits of the same system can land on different conclusions. Making the scoring deterministic and tied to a versioned policy object means an assessment can be rerun months later against a new snapshot and produce a comparable number. That is exactly what auditors and regulators ask for and rarely get.
One catch the paper is upfront about. Hardening moves like SBOMs and scanning gates do push the feasibility score down, but if the one control that actually blocks a given attack technique is missing from the evidence, no amount of unrelated hardening fixes that number.