A new training method fixes a subtle data problem that has been quietly warping how AI image generators learn what people like.
Researchers identify a flaw in Diffusion Direct Preference Optimization (DPO), the technique used to align image-generation models with human taste. Existing datasets label each image pair simply as winner or loser, even though real preferences span multiple dimensions: an image can nail aesthetics while botching fine detail or missing what the prompt actually asked for. Collapsing that nuance into one binary label creates what the researchers call label noise, and they show mathematically that it sends conflicting gradient signals during training. Their fix, called Semi-DPO, splits preference pairs into a clean set where signals agree and a noisy set where they conflict, trains first on the clean data, then uses that model to generate pseudo-labels that iteratively clean up the rest.
This matters because preference-based alignment keeps getting treated as a data-volume problem, when the labels themselves may be the real bottleneck. Semi-DPO reportedly reached state-of-the-art alignment results without extra human annotation or an explicit reward model, meaning better statistics beat more raters here.
It is a narrow fix for a narrow technique. But if noisy binary labels are corrupting DPO training, the same blind spot likely lurks in other RLHF-style pipelines built on the same winner-loser assumption.