An AI hardware-security benchmark just caught its own test setup lying to it.
A new arXiv preprint (arXiv:2609.19844) introduces SecTB-RTL, a framework for auditing AI-generated chip verification plans against 124 hand-authored hardware-security regressions spread across 31 tasks. A deterministic, non-AI baseline killed 36, 75, and 78 of the framework's test mutants as resource limits increased - a mutation-testing sanity check, and a separate metric from the 124 authored regressions themselves. The real trouble showed up once the AI model actually ran: a first confirmatory run failed before execution because the provider rejected its response schema. After a blind, schema-only fix, a second frozen run completed 1,860 calls - the provider accepted 1,857 of them, but only nine survived the full production semantic validator.
That gap between accepted and actually worked is the paper's real finding, and it's an infrastructure warning rather than a capability score. The researchers declined to report any prompt-effect estimate at all, instead preserving the run as an instrument-validation incident so infrastructure failures don't get mistaken for model failures.
It's a useful check on AI benchmarking generally: a provider's schema acceptance tells you the plumbing works, not that the underlying plan is any good.