A new arXiv paper tries to put numbers on a question AI labs have mostly argued about with vibes: when should you train a model to behave, and when should you just build better filters to catch it misbehaving?
Researchers built a mathematical model that treats AI safety as a budget-allocation problem, splitting resources between "character shaping" (training-time methods like RLHF and Constitutional AI that try to instill good behavior) and "rule enforcement" (output filters and safety classifiers that catch bad outputs after the fact). The model factors in filters getting weaker at scale, shared failure modes across systems, and what the authors call "character fragility," the risk that trained-in behavior collapses when a model meets something it has not seen before. Testing optimistic, moderate, and pessimistic scenarios, they found the optimal split tilts toward character shaping as deployment scale grows, though the size of that shift varies widely, from a negligible 0.01 nudge to a much larger 0.21 swing depending on assumptions. The single biggest factor in the math was not scale at all. It was how reliably trained behavior holds up under new conditions, a variable that moved the optimal balance by 0.50, dwarfing every other factor tested, including filter quality and shared failure risk.
This is a useful corrective to a debate that often treats more RLHF or more filtering as inherently safer by default. The paper's math says the deciding factor is not deployment size, it is whether character training actually generalizes to situations it was never trained on, a property nobody currently knows how to measure reliably in production systems.
The paper's real contribution is not a formula for the perfect safety mix. It is a reminder that the industry is optimizing an allocation problem while still guessing at one of its most important inputs.