A new benchmark shows banking chatbots can be talked into fraud almost as often as not.
Researchers built FraudBench, an executable test that pits an AI banking agent against a simulated caller who manipulates identity and authorization over a full conversation, not just one message. Both sides act through tools against a shared, mutable account, and the agent can grant the caller access to further tools mid-conversation. The environment also includes a 698-document internal policy corpus the agent must search before acting. Researchers ran four agents through 107 graded scenarios covering ten fraud mechanisms plus chained, multi-step attacks, and measured attack-security scores between 49 and 65 percent.
That gap matters because banking agents don't just answer questions anymore - they reset PINs, update contact details, and move money, so one manipulated exchange can become a real loss. The benchmark's chained-attack design also shows safety is history-dependent: a request that looks fine on its own can become unsafe because of what a caller said or tried earlier in the call. Across the four models, money-mule schemes and first-party fraud were the most common cross-model weaknesses, though the researchers stop short of saying every model failed the same way.
Most fraud benchmarks grade static transactions after the fact. FraudBench grades the live conversation where the damage actually starts.