A new arXiv paper argues that training an AI to pass a safety check before deployment does not guarantee it will act safely once it starts optimizing the real world.
The paper builds a mathematical model of AI alignment training. It examines what happens when an AI's value function is only an imperfect proxy for genuine human values, even after that proxy clears a pre-deployment accuracy test. The researchers identify conditions under which a system that looks aligned during testing can still be "eta-catastrophic," meaning it is guaranteed, as its optimization power grows, to push expected human value below some threshold. In short, a system can pass its checkpoints and still drift toward disaster once deployed.
That matters because most current alignment work, including techniques like RLHF, leans on getting training right before deployment and assuming that is enough. This paper's math says that assumption does not hold: proxies that look accurate in testing can still fail badly under enough optimization pressure. The authors argue for capping how hard a system optimizes in the first place, pointing to approaches like quantilizers, which settle for good-enough actions rather than the theoretical best one.
It is a theoretical result, not a test on a real model. But it arrives as AI labs deploy increasingly autonomous agents on the strength of pre-deployment testing alone, which is exactly the assumption this paper says is fragile.