Aligning AI models exacts a price beyond raw performance — and a paper out this week proposes a cheap fix.
Researchers studying the "alignment tax" found that safety-tuning a model does not merely dent task accuracy — it also makes models overconfident and less diverse in their outputs. Their proposed remedy is model merging: interpolating between a model's weights before and after alignment training. The technique is post-hoc, meaning it requires no retraining. More notably, the team found that the right interpolation point is not a simple compromise. Certain blends outperform both the unaligned and the aligned parent models on accuracy while recovering most of the calibration that alignment stripped away.
Calibration — how well a model's stated confidence tracks its actual correctness — is less visible than benchmark scores but arguably more important in production. An overconfident model that is wrong 30% of the time but expresses near-certainty is more dangerous than a model that simply scores lower. If weight interpolation can restore calibration without a tuning run, it becomes a practical lever for teams already deploying aligned models who cannot afford another training cycle.
Weight averaging between model checkpoints is an old trick in deep learning, but applying it to the alignment-before/after axis is a relatively underexplored angle. The approach sidesteps the cost of methods like constitutional AI or reinforcement learning from human feedback fine-tuning, though it also does nothing to strengthen the alignment itself — teams hunting for Pareto wins here should note they are recovering reliability, not adding safety.