Nubank put its customer-service chatbots through simulated conversations before shipping them to real customers.
Researchers built a hypothesis-driven simulation workflow using a tool called Snowglobe, testing it on Nubank's Card Delivery agent and its successor, Card Management - the bank's highest-volume chat-support system in Brazil. Synthetic customers react to the agent's replies while simulated tool outputs stand in for production backends, letting testers run multi-step conversations without touching real systems. Across four deployed versions, scores from the simulation closely tracked scores measured in production. In one live A/B test, iterating on the agent inside the simulator lifted transactional net promoter score by 36.69 points.
A separate exercise pushed the approach further: Nubank screened over 16,000 simulated conversations to compare open-weight model configurations before picking a winner. That model went on to raise self-service rate by 8.82 percentage points in production, the highest level Nubank has recorded - though satisfaction scores in that particular test didn't move.
For an industry that answers to regulators as well as customers, this matters because live experimentation is expensive to get wrong. A failed test isn't just a bad metric - it's a real customer stuck with an unhelpful bank chatbot. Simulation lets Nubank try far more configurations than it could ever justify testing on live traffic.
Two experiments, two different wins, and no single test where both satisfaction and self-service rose together - a reminder that simulated screening picks good candidates, it doesn't guarantee agents that are both faster and more liked.