A new arXiv paper argues that acing SWE-bench or LiveCodeBench does not mean a model can actually code well.
Researchers built a Django-based case study suite to test whether benchmark gains generalize. They evaluated foundation models and checkpoints post-trained on SWE-bench trajectories, then checked performance on their own tasks and on LiveCodeBench. The result: benchmark rankings often did not carry over. Models fine-tuned on individual Django modalities showed little to no transfer to other tasks, and optimizing for SWE-bench produced limited or no improvement elsewhere.
This matters because SWE-bench and LiveCodeBench scores show up constantly in model cards and launch blog posts, treated as proof of general coding skill. If a high score mostly reflects narrow, task-specific tuning, then engineers picking a model based on a leaderboard number could be optimizing for the wrong thing entirely. The paper calls for differentiated evaluation instead: holistic testing for frontier models, multi-task suites for research, and human-in-the-loop review for narrow applications.
Benchmark gaming is an old problem in machine learning, but it is a newer one in the era of models marketed on single headline scores. The paper's suggestion of a capability taxonomy with ongoing maintenance, rather than one-off leaderboards, is a sensible fix. Whether any lab has an incentive to adopt it before a competitor forces the issue is a separate question.