A team of researchers just had large language models write, and then grade, their own training data for fixing code.
The approach works in two phases. First, several state-of-the-art LLMs generated roughly 30,000 paired examples of buggy and fixed code, spanning 12 programming languages and 13 categories of bugs. Then those same models cross-evaluated each other's output against five criteria: correctness, code quality, security, performance, and completeness, keeping only the samples that passed muster. On the VulRepair benchmark, a repair system trained on this filtered dataset hit a 17.18% Top@1 perfect-prediction rate, up from an 11.68% baseline (a 47% relative improvement), and 23.00% Top@5, up from 18.88% (a 22% relative improvement). The results held up under ANOVA and Tukey's HSD statistical testing, and the best configurations beat existing systems while using a cheaper decoding strategy.
The real story here is about data, not code. Automated program repair has long been bottlenecked by a shortage of high-quality, diverse bug examples to train on. Using LLMs to both manufacture and referee that training data is a workaround that could generalize well beyond bug fixing, to any software engineering task starved of labeled examples.
It is also a closed loop worth watching skeptically: the same class of model is doing the grading and the generating. If the graders share blind spots with the generators, quality filtering may just be teaching the system to prefer its own kind of mistake.