A new benchmark suggests much of what looks like an AI agent's hardware expertise is really just efficient guessing dressed up in engineering jargon.
Researchers built AutoTuring, which gives the same AI agent the same 15-dimensional accelerator design space twice: once labeled with real architectural terms and simulator counters, once rewritten as anonymous numbers between 0 and 1. The evaluator, the legal search space, and the best possible outcomes stay identical between runs. Only the framing changes. On a set of nine FP16 matrix-multiplication (GEMM) kernels, the agent that could see it was tuning cache sizes and pipeline depths beat a modeled Nvidia H200 GPU configuration by 5.4%, beat its own blindfolded version by 12.3%, and needed 70.1% fewer simulator calls to get there.
Those numbers measure two different things, not one. The 5.4% and 12.3% figures describe how good the final chip design is; the 70.1% figure describes how much less trial-and-error it took to find it. The more striking result is what erases the performance gap: giving the blindfolded agent a critic loop that reviews and revises its own guesses recovers most of the difference, while the same critique does nothing for the informed agent, meaning architectural knowledge and structured feedback act as substitutes, not complements.
The authors call this preliminary, based on five to six runs per condition on a single simulated accelerator, which is thin enough that the percentages should be read as a direction rather than a verdict.