Training an LLM to write better code through alignment is messier than the AI labs would like to admit.
Researchers ran an empirical study on five state-of-the-art large language models, applying two reward-free alignment techniques — Direct Preference Optimization (DPO) and BoNBoN — to both pretrained and instruction-tuned model variants. They measured results across four functional benchmarks (HumanEval+, MBPP+, EvalPerf, EvoEval) and a code quality benchmark called CODAL that covers readability, style, and maintainability. The preference data used to train each model came from the SelfCodeAlign pipeline, which generates accepted and rejected code pairs without requiring separate reward scores.
The core tension: starting alignment from a raw pretrained model produces bigger relative gains, but those models were already behind instruction-tuned versions on raw accuracy. Starting from a fine-tuned model yields smaller improvements — or in some cases, outright regression. That puts practitioners in an awkward spot where neither path is clearly dominant, and the right choice depends on which trade-off hurts less.
Most alignment research targets non-coding tasks, so the field has been extrapolating to code largely on faith. This study is an early sanity check — and the answer is that the assumptions don't hold cleanly. If your code LLM is already fine-tuned, layering alignment on top may cost you more than it buys.