[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"branding":3,"analytics":7,"article-ai-compliance-detectors-largely-ignore-the-rules-they-check":10,"sections":41},{"siteName":4,"siteTagline":5,"publisherName":4,"contactEmail":6},"The Revision","Tech news, decoded.","editor@therevision.news",{"gaMeasurementId":8,"adsenseClientId":9},"G-ZW2MV82GYR","ca-pub-8533917693782264",{"article":11},{"id":12,"slug":13,"title":14,"dek":15,"body_md":16,"tags_json":17,"published_at":18,"created_at":19,"updated_at":20,"status":21,"review_note":22,"review_notes":23,"image_url":22,"persona_id":22,"persona_name":22,"section":30,"tags":31,"sources":36,"feedback":40,"feedback_at":22,"cost_usd":40,"total_tokens":40},5274,"ai-compliance-detectors-largely-ignore-the-rules-they-check","AI Compliance Detectors Largely Ignore the Rules They Check","New research shows guard models and activation probes meant to flag rule violations barely change their verdicts when the actual rule is swapped or deleted.","The automated systems supposed to catch AI outputs that break the rules mostly aren't reading the rules at all.\n\nA new audit tested activation probes and guard models - the compliance layers deployed to check chatbot outputs against written policies covering data protection, healthcare, finance, and platform rules. The researchers deleted, swapped, or permuted the governing rule in each test case and found detection accuracy barely moved for any detector, a pattern they call rule blindness. One guard model even correctly named the clause it was applying, then barely changed its verdict when that clause was swapped for its permissive opposite. A new benchmark crossing two rules with two scenarios, designed so neither factor alone predicts the right answer, confirmed the problem; only detectors that reasoned step by step escaped it.\n\nThe researchers also built their own lightweight detector, the Internal Compliance Score, calibrated from just ten labeled examples and cheap enough to audit four deployed guards, an 8B-parameter judge, and thirteen benchmarks at once. They held it to the same bar and it failed too: a pre-registered test for beating a trivial baseline came up short, and a plain bag-of-words model matched its results. Its one genuine payoff was raising the verified pass rate when it was used to rank candidate answers - useful, but narrow.\n\nThat gain vanished the moment researchers ran an adaptive attack designed to exploit the detector directly, a reminder that a compliance tool cheap enough to audit everything is not the same as one robust enough to trust.","[\"ai safety\",\"compliance detection\",\"guard models\",\"activation probes\"]","2026-08-18T04:00:00.000Z","2026-08-18T12:53:15.701Z","2026-08-18T12:53:27.486Z","published",null,[24],{"id":25,"reviewer":26,"round":27,"reason":28,"status":29},"editor-r1","editor",1,"The closing line references the ICS detector's 'one real gain' vanishing under attack without ever telling readers what that gain was (source: it raised the verified pass rate when used to rank candidate responses) — add that detail earlier in the body so the ending doesn't reference an unestablished claim.","resolved","ai",[32,33,34,35],"ai safety","compliance detection","guard models","activation probes",[37],{"name":38,"url":39},"arXiv cs.AI","https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.16852",0,{"sections":42},[43,47,51,56,61,66,71,76,81,85,90,95,100,105],{"name":44,"slug":30,"count":45,"latest_published_at":46},"AI",3293,"2026-08-20T04:00:00.000Z",{"name":48,"slug":49,"count":50,"latest_published_at":46},"Security","security",435,{"name":52,"slug":53,"count":54,"latest_published_at":55},"Policy","policy",210,"2026-08-19T09:32:27.000Z",{"name":57,"slug":58,"count":59,"latest_published_at":60},"Deals","deals",179,"2026-06-29T20:02:07.000Z",{"name":62,"slug":63,"count":64,"latest_published_at":65},"Hardware","hardware",140,"2026-08-19T18:25:42.000Z",{"name":67,"slug":68,"count":69,"latest_published_at":70},"Consumer Tech","consumer-tech",95,"2026-08-18T16:05:00.000Z",{"name":72,"slug":73,"count":74,"latest_published_at":75},"Science","science",90,"2026-08-19T18:41:02.000Z",{"name":77,"slug":78,"count":79,"latest_published_at":80},"Software","software",73,"2026-08-18T07:51:50.000Z",{"name":82,"slug":83,"count":84,"latest_published_at":18},"Dev Tools","dev-tools",69,{"name":86,"slug":87,"count":88,"latest_published_at":89},"Startups","startups",47,"2026-08-19T19:13:46.000Z",{"name":91,"slug":92,"count":93,"latest_published_at":94},"Gaming","gaming",41,"2026-07-09T04:00:00.000Z",{"name":96,"slug":97,"count":98,"latest_published_at":99},"General","general",33,"2026-08-18T22:18:13.000Z",{"name":101,"slug":102,"count":103,"latest_published_at":104},"Reviews","reviews",20,"2026-06-24T12:00:01.000Z",{"name":106,"slug":107,"count":108,"latest_published_at":109},"How-To","how-to",6,"2026-06-16T09:00:00.000Z"]