Banks are handing real decisions to AI agents that often can't explain, or even repeat, what they just did.
Researchers tested nine model versions, from a 3-billion-parameter local model to a commercial frontier system, across three studies simulating financial-services use. They found that when a provider updates its model, the record of past financial actions can shift underneath it, and the frontier model blocks the exact settings, temperature, top_p, top_k, and random seed, that would let anyone replay a decision. Under the strictest controls each system allowed, the local model reproduced all 320 test runs, while hosted models matched 319 of 320 and 959 of 960. A second study found the orchestration layer, the software that coordinates multiple models and tools, changes outcomes on its own, and no execution repeated exactly across any configuration or scale. A third study showed two deterministic credit-scoring model versions could each perfectly reproduce their own current decisions, but the newer version could not recreate a decision the older one had made.
Financial regulators generally expect firms to reconstruct and defend automated decisions after the fact. This research suggests that expectation collapses the moment a provider pushes a model update or tweaks its orchestration logic, even when nothing about the underlying policy has changed. The authors argue reproducibility should be treated as a continuously monitored governance profile, not a one-time assumption baked into a compliance checklist.
The irony is hard to miss: the more capable the model, the less anyone outside its provider can prove what it actually did.