Diffusion models are good at classifying common images — and surprisingly bad at everything else.
A new paper introduces MiPO (Minority Preference Optimization), a fine-tuning approach that targets the core weakness of diffusion-based classifiers: they learn from high-density regions of training data, which means familiar-looking images get recognized accurately while rare or atypical ones fall through the cracks. The researchers show that minority sampling — a technique previously studied mostly for generating unusual images — also directly improves a model's ability to perceive those underrepresented categories. MiPO works by generating candidate images, rewarding outputs that cover low-density regions of the data distribution, and then optimizing the model using LoRA and Group Relative Policy Optimization. No extra image datasets, no external models, and no external reward signals are required — just arbitrary caption data.
This matters because diffusion classifiers have become a quiet alternative to discriminative models for zero-shot classification, and their accuracy gap on minority classes is a practical problem in any deployment touching long-tail distributions — medical imaging, rare species identification, low-volume product categories. Closing that gap without needing more labeled data is a meaningful step.
The approach was evaluated across five standard datasets, though the paper does not claim it erases the majority-minority gap entirely — just that it reliably narrows it. Whether the gains hold outside tightly controlled benchmarks is the question any real-world deployment will have to answer.