A new study shows that teaching code-generating AI to second-guess itself usually makes things worse, unless it actually runs the code to check.
Researchers tested five ways to estimate how confident a code-generating model is in its own answer, including asking the model to state its confidence and measuring how "surprised" it was by its own tokens. They tried three small AI models on two standard coding benchmarks, HumanEval and BigCodeBench. Only one method, called P(True) with multiple samples, correlated well with whether the code was actually correct. When they used these confidence signals to decide whether to regenerate or adjust code, accuracy got worse in most cases, dropping 3 to 10 percentage points in five of six setups.
The one approach that consistently helped was not confidence estimation at all: actually running the generated code against tests before deciding whether to keep or redo it. That method boosted accuracy by 6 to 26 points on HumanEval and 8 to 20 points on BigCodeBench, with bigger gains for weaker starting models. For teams building AI coding tools, the takeaway is blunt: don't trust a model's gut feeling about its own code, trust the test suite.
Confidence scores make for a nice progress bar, but only actual execution tells you if the code works.