A fleet of AI agents can each pass every individual safety check and still blow past a company's total tolerance for irreversible mistakes.
Researchers propose an "irreversibility budget": a running account of residual risk that a trusted runtime tracks for a company across every agent, workflow, and team it operates. Each time an agent moves money, deploys code, deletes data, or discloses information, the runtime deducts the potential loss from that shared account. Once the total would exceed the company's risk limit, the runtime blocks the next irreversible action, even if that specific agent's own gate would have approved it. In controlled tests, checking each agent's actions in isolation - the current norm - let a fleet overdraw its risk limit by up to 48 times before anything caught it; the budget-tracking approach kept every run inside the limit.
The gap here isn't a rogue agent ignoring the rules. It's a swarm of individually compliant agents each taking a small, approved risk that adds up to real damage - the same blind spot that let coordinated fraud slip past transaction-by-transaction checks until banks started watching spend across accounts instead of one swipe at a time.
The hard part is pricing the risk correctly. The paper admits that valuing an action - when the input comes from agents that may misdeclare it, and effects are correlated with each other - is still unsolved. So this is a solid accounting model without a trustworthy price tag yet.