A new alignment technique aims to stop AI models from being safe in aggregate while still failing badly on specific categories of harm.
Researchers describe Cat-DPO, an extension of direct preference optimization (DPO) - the standard method for teaching large language models to refuse harmful requests - in a paper posted to arXiv. Instead of scoring every category of harm against one global safety target, Cat-DPO gives each harm category its own adaptive margin. That margin tightens automatically when a model keeps producing unsafe answers in a category, and loosens once the model catches up, so training pressure follows wherever the model is actually failing. Tested across two LLM backbones against six existing preference-learning baselines, Cat-DPO improved combined helpfulness and harmlessness scores while shrinking the spread between a model's best- and worst-performing safety categories.
That spread is the real problem with current safety training. A model can post strong scores on standard safety benchmarks, which tend to average results across many categories, while remaining exploitable on whichever category the benchmark under-weights. Cat-DPO's category-by-category accounting is a more honest way to measure whether alignment actually holds up, not just where it is easiest to improve.
It is a training-time fix validated on the categories the researchers chose to measure, not a guarantee against whatever harm category an attacker decides to try next.