AI agents are still poor substitutes for engineers who know how to make code fast.
Researchers have released SWE-fficiency, a benchmark that tasks AI agents with optimizing performance in large, real-world repositories — numpy, pandas, scipy, and six other data-science and HPC codebases. Unlike most coding benchmarks that ask models to fix bugs, SWE-fficiency focuses on the harder question of how to make code faster. Each of 498 tasks gives an agent a full codebase and a slow workload; the agent must find the bottleneck, produce a patch, and pass the same unit tests that a human expert's solution passes. Speedup targets are derived from actual GitHub pull requests, validated with static analysis and execution tooling.
The results are deflating for anyone pitching AI as a drop-in replacement for senior engineers. State-of-the-art agents averaged less than 0.23x the speedup a human expert achieved on the same task — meaning the best current models deliver under a quarter of the performance gain a skilled engineer would. The benchmark exposes three specific failure modes: agents struggle to localize where slowdowns actually occur, they have trouble reasoning about how execution flows across multiple functions, and they frequently break correctness when they do attempt an edit.
That last failure mode matters most. A patch that makes code faster but wrong is worse than no patch at all. The gap between fixing a bug (the thing most benchmarks measure) and genuinely improving performance under real workloads turns out to be wide — and current agents are still on the wrong side of it.