A research team has released MegaBugFix, a large-scale benchmark designed to stress-test how well large language models actually fix code bugs.
Most existing bugfix benchmarks are small and cover a narrow slice of bug types, which means models can look capable without being capable. MegaBugFix takes a different approach: it uses an LLM to inject bugs into correct Python programs by generating diffs — structured representations of code changes — rather than swapping in simple text mutations. The result is 12,629 buggy programs that more closely resemble the messy, context-dependent bugs developers encounter in the real world. The researchers tested 13 open-weight models against MegaBugFix and found that every one scored lower than on existing benchmarks.
That performance gap matters because benchmark scores shape which models get adopted for coding tools, code review assistants, and automated repair pipelines. If the benchmarks are too easy or too narrow, developers end up deploying models that underperform on real codebases. MegaBugFix's diff-based injection method also sidesteps a known failure mode of LLM-based mutation: generating trivially simple bugs that any model can catch.
The benchmark joins a growing pile of evidence that the coding ability leaderboard looks friendlier than the underlying reality — and that diff-aware evaluation may be the closer proxy for what actually happens when a model meets a pull request.