A new alignment method skips the gradient math that trips up today's most popular way of tuning chatbots on human preferences.
Researchers propose ComPO, short for Comparison-based Preference Optimization, a "zeroth-order" method - meaning it never computes a differentiable loss from a preference pair, and instead uses comparison oracles to figure out which direction to nudge the model. It targets a known flaw in Direct Preference Optimization (DPO): when two responses in a training pair are nearly equally likely, DPO can accidentally push down the probability of the preferred response too, a failure mode called likelihood displacement. The team built an offline version with a mathematical convergence proof and an online version that adds KL-based guardrails so the model does not drift too far from its starting point. They tested both on five open-source model families - Mistral, Llama, Gemma-2, Qwen3, and Gemma-3 - and reported better length-controlled win rates than existing direct alignment methods, backed by pair-level diagnostics.
DPO and its variants are now the default way most labs fine-tune models on preference data, prized because they skip the separate reward model that older RLHF pipelines required. Likelihood displacement is a documented crack in that shortcut, so a gradient-free fix that avoids reintroducing a reward model would patch plumbing nearly every instruction-tuned model depends on.
The paper leans on convergence guarantees and diagnostics rather than leaderboard bragging, which is the right instinct this early - the real test is whether ComPO holds up outside five research checkpoints and inside an actual production fine-tuning run.