Researchers have found that AIs trained to prefer safe choices in small-stakes situations carry that caution into scenarios with astronomically higher stakes.
A team introduced RiskAverseOOD, a benchmark for testing whether risk aversion learned on low-stakes gambles holds up when the stakes jump by 98 orders of magnitude. They trained Qwen3-8B using several methods — supervised fine-tuning and tie training achieved around 70% "Cooperate" rates from a baseline of 2%, DPO reached 52%, and activation steering hit 39%. A fine-tuned reward model correctly ranked risk-averse reasoning above alternatives in 99.6% of pairwise comparisons. The effects replicated across Qwen3 variants at 1.7B and 14B parameters, and separately held up across different model families including Gemma-3-12B-IT and Llama-3.1-8B-Instruct.
The safety argument here is simple: a misaligned AI that still prefers low-risk strategies will tend toward cooperation over rebellion, limiting the worst-case damage. If risk aversion can be reliably baked in at training time, it becomes a potential failsafe that doesn't require perfect alignment. The researchers are clear that results aren't consistent enough yet to call it a genuine safety guarantee.
Until that consistency problem is solved, "partial generalization" is just another way of saying the failsafe sometimes works.