[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"branding":3,"analytics":7,"article-ai-chip-verification-benchmark-exposes-a-provider-schema-trap":10,"sections":41},{"siteName":4,"siteTagline":5,"publisherName":4,"contactEmail":6},"The Revision","Tech news, decoded.","editor@therevision.news",{"gaMeasurementId":8,"adsenseClientId":9},"G-ZW2MV82GYR","ca-pub-8533917693782264",{"article":11},{"id":12,"slug":13,"title":14,"dek":15,"body_md":16,"tags_json":17,"published_at":18,"created_at":19,"updated_at":20,"status":21,"review_note":22,"review_notes":23,"image_url":22,"persona_id":22,"persona_name":22,"section":30,"tags":31,"sources":36,"feedback":40,"feedback_at":22,"cost_usd":40,"total_tokens":40},7008,"ai-chip-verification-benchmark-exposes-a-provider-schema-trap","AI Chip Verification Benchmark Exposes a Provider Schema Trap","A hardware-security benchmark found an AI verification run where providers accepted 1,857 responses but only nine passed validation.","An AI hardware-security benchmark just caught its own test setup lying to it.\n\nA new arXiv preprint (arXiv:2609.19844) introduces SecTB-RTL, a framework for auditing AI-generated chip verification plans against 124 hand-authored hardware-security regressions spread across 31 tasks. A deterministic, non-AI baseline killed 36, 75, and 78 of the framework's test mutants as resource limits increased - a mutation-testing sanity check, and a separate metric from the 124 authored regressions themselves. The real trouble showed up once the AI model actually ran: a first confirmatory run failed before execution because the provider rejected its response schema. After a blind, schema-only fix, a second frozen run completed 1,860 calls - the provider accepted 1,857 of them, but only nine survived the full production semantic validator.\n\nThat gap between accepted and actually worked is the paper's real finding, and it's an infrastructure warning rather than a capability score. The researchers declined to report any prompt-effect estimate at all, instead preserving the run as an instrument-validation incident so infrastructure failures don't get mistaken for model failures.\n\nIt's a useful check on AI benchmarking generally: a provider's schema acceptance tells you the plumbing works, not that the underlying plan is any good.","[\"ai evaluation\",\"hardware security\",\"chip verification\",\"research\"]","2026-09-18T04:00:00.000Z","2026-09-19T06:15:29.599Z","2026-09-19T06:15:41.512Z","published",null,[24],{"id":25,"reviewer":26,"round":27,"reason":28,"status":29},"editor-r1","editor",1,"Attribute the findings to their actual source (name it as an arXiv preprint and cite arXiv:2609.19844) since 'researchers' is currently unnamed with no institution or link, and fix the baseline sentence, which claims the deterministic baseline 'killed 36, 75, and 78 of those regressions' when the source says it killed mutants, a distinct mutation-testing concept from the 124 authored regressions.","resolved","ai",[32,33,34,35],"ai evaluation","hardware security","chip verification","research",[37],{"name":38,"url":39},"arXiv cs.AI","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.19844",0,{"sections":42},[43,47,52,57,62,67,72,77,81,86,91,96,101,106],{"name":44,"slug":30,"count":45,"latest_published_at":46},"AI",4114,"2026-09-19T18:33:46.000Z",{"name":48,"slug":49,"count":50,"latest_published_at":51},"Security","security",673,"2026-09-19T17:30:00.000Z",{"name":53,"slug":54,"count":55,"latest_published_at":56},"Policy","policy",345,"2026-09-19T19:22:20.000Z",{"name":58,"slug":59,"count":60,"latest_published_at":61},"Deals","deals",179,"2026-06-29T20:02:07.000Z",{"name":63,"slug":64,"count":65,"latest_published_at":66},"Hardware","hardware",156,"2026-09-19T11:00:00.000Z",{"name":68,"slug":69,"count":70,"latest_published_at":71},"Science","science",129,"2026-09-19T19:45:00.000Z",{"name":73,"slug":74,"count":75,"latest_published_at":76},"Consumer Tech","consumer-tech",99,"2026-09-09T17:27:33.000Z",{"name":78,"slug":79,"count":80,"latest_published_at":18},"Dev Tools","dev-tools",78,{"name":82,"slug":83,"count":84,"latest_published_at":85},"Software","software",75,"2026-09-10T20:41:21.000Z",{"name":87,"slug":88,"count":89,"latest_published_at":90},"Startups","startups",55,"2026-09-09T23:14:29.000Z",{"name":92,"slug":93,"count":94,"latest_published_at":95},"Gaming","gaming",43,"2026-09-10T12:18:06.000Z",{"name":97,"slug":98,"count":99,"latest_published_at":100},"General","general",42,"2026-09-18T22:35:10.000Z",{"name":102,"slug":103,"count":104,"latest_published_at":105},"Reviews","reviews",20,"2026-06-24T12:00:01.000Z",{"name":107,"slug":108,"count":109,"latest_published_at":110},"How-To","how-to",6,"2026-06-16T09:00:00.000Z"]