Standard AI safety training has a blind spot: it optimizes for average behavior, which means rare disasters can slip through.
Researchers have proposed Risk-sensitive Alignment via Dominance (RAD), a framework that swaps out the expected-cost constraints used in standard Safe RLHF for something more rigorous. Instead of asking "what is the average harm this model causes?", RAD asks whether the model's full distribution of harmful outputs is better than a reference policy's distribution — a concept called First-Order Stochastic Dominance. The team operationalizes this using Optimal Transport math, with entropic regularization and Sinkhorn iterations making the whole thing differentiable and trainable end-to-end. They also introduce quantile-weighted versions of the constraint that give a principled dial for tuning how much the model cares about tail risk versus typical behavior.
The practical upshot is that a model trained with RAD is harder to break in unusual or out-of-distribution scenarios — exactly the cases where aligned models tend to embarrass their makers publicly. Expected-value safety is a reasonable first approximation, but heavy-tailed cost distributions mean the average can look fine right up until a rare input causes a serious failure; RAD is designed to close that gap without sacrificing helpfulness.
The approach draws on actuarial and financial risk theory — spectral risk measures are standard tools in portfolio management — which raises a fair question about whether the AI safety community has been leaving established statistical machinery on the table.