A research benchmark called Metal-Sci reveals that LLMs can write fast Apple Silicon compute kernels, but speed alone is a misleading scorecard.
Metal-Sci presents ten tasks drawn from scientific computing — stencils, n-body simulations, Boltzmann solvers, molecular dynamics, PDE pipelines, and FFT — each targeting Apple's Metal GPU API. Researchers paired each task with a CPU reference and a roofline-anchored fitness function, then set Claude Opus 4.7, Gemini 3.1 Pro, and GPT 5.5 loose inside a simple evolutionary loop: write a kernel, compile it, score it, repeat. Across that in-distribution loop, self-speedups ranged from 1.00x to 10.7x depending on the task and model. That headline number sounds impressive until you read the fine print.
The more interesting result is the held-out gate: a scoring function evaluated once at end-of-run on a problem size the model never saw during search. It caught two failures that the in-distribution score masked entirely. GPT's best FFT3D kernel hit 2.95x speedup on training sizes but collapsed to 0.23x on a 256-cubed held-out cube. Claude's top HMC kernel was faster but returned wrong samples at unseen dimensions — a correctness failure hiding behind a speed win. Those are exactly the failure modes that matter when someone deploys auto-generated compute code in production.
The benchmark's real contribution is methodological: cheap held-out evaluation as a mechanical oversight primitive on automatic code-generation loops. It is a modest but useful check on a workflow that the AI-coding hype cycle tends to present as solved.