[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"branding":3,"analytics":7,"article-nubank-simulates-customers-to-test-support-chatbots-before-launch":10,"sections":41},{"siteName":4,"siteTagline":5,"publisherName":4,"contactEmail":6},"The Revision","Tech news, decoded.","editor@therevision.news",{"gaMeasurementId":8,"adsenseClientId":9},"G-ZW2MV82GYR","ca-pub-8533917693782264",{"article":11},{"id":12,"slug":13,"title":14,"dek":15,"body_md":16,"tags_json":17,"published_at":18,"created_at":19,"updated_at":20,"status":21,"review_note":22,"review_notes":23,"image_url":22,"persona_id":22,"persona_name":22,"section":30,"tags":31,"sources":36,"feedback":40,"feedback_at":22,"cost_usd":40,"total_tokens":40},7780,"nubank-simulates-customers-to-test-support-chatbots-before-launch","Nubank Simulates Customers to Test Support Chatbots Before Launch","Nubank tested chat support agents on simulated customers, boosting satisfaction in one trial and self-service rates in another, not both at once.","Nubank put its customer-service chatbots through simulated conversations before shipping them to real customers.\n\nResearchers built a hypothesis-driven simulation workflow using a tool called Snowglobe, testing it on Nubank's Card Delivery agent and its successor, Card Management - the bank's highest-volume chat-support system in Brazil. Synthetic customers react to the agent's replies while simulated tool outputs stand in for production backends, letting testers run multi-step conversations without touching real systems. Across four deployed versions, scores from the simulation closely tracked scores measured in production. In one live A\u002FB test, iterating on the agent inside the simulator lifted transactional net promoter score by 36.69 points.\n\nA separate exercise pushed the approach further: Nubank screened over 16,000 simulated conversations to compare open-weight model configurations before picking a winner. That model went on to raise self-service rate by 8.82 percentage points in production, the highest level Nubank has recorded - though satisfaction scores in that particular test didn't move.\n\nFor an industry that answers to regulators as well as customers, this matters because live experimentation is expensive to get wrong. A failed test isn't just a bad metric - it's a real customer stuck with an unhelpful bank chatbot. Simulation lets Nubank try far more configurations than it could ever justify testing on live traffic.\n\nTwo experiments, two different wins, and no single test where both satisfaction and self-service rose together - a reminder that simulated screening picks good candidates, it doesn't guarantee agents that are both faster and more liked.","[\"ai agents\",\"customer support\",\"simulation testing\",\"fintech\"]","2026-09-25T04:00:00.000Z","2026-09-25T21:58:32.435Z","2026-09-25T21:58:38.516Z","published",null,[24],{"id":25,"reviewer":26,"round":27,"reason":28,"status":29},"editor-r1","editor",1,"Fix the dek: it claims Nubank boosted both self-service rates and satisfaction, but the body shows these came from two separate experiments and the self-service gain explicitly had no significant satisfaction change — the dek overclaims a combined win the evidence doesn't support.","resolved","ai",[32,33,34,35],"ai agents","customer support","simulation testing","fintech",[37],{"name":38,"url":39},"arXiv cs.AI","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.30137",0,{"sections":42},[43,47,52,57,62,67,71,76,81,86,91,96,100,105],{"name":44,"slug":30,"count":45,"latest_published_at":46},"AI",4635,"2026-09-26T23:42:06.000Z",{"name":48,"slug":49,"count":50,"latest_published_at":51},"Security","security",751,"2026-09-26T12:00:00.000Z",{"name":53,"slug":54,"count":55,"latest_published_at":56},"Policy","policy",396,"2026-09-26T18:45:15.000Z",{"name":58,"slug":59,"count":60,"latest_published_at":61},"Deals","deals",260,"2026-09-26T09:00:00.000Z",{"name":63,"slug":64,"count":65,"latest_published_at":66},"Hardware","hardware",186,"2026-09-26T17:26:54.000Z",{"name":68,"slug":69,"count":70,"latest_published_at":61},"Science","science",144,{"name":72,"slug":73,"count":74,"latest_published_at":75},"Consumer Tech","consumer-tech",135,"2026-09-26T14:30:00.000Z",{"name":77,"slug":78,"count":79,"latest_published_at":80},"Software","software",91,"2026-09-25T20:55:00.000Z",{"name":82,"slug":83,"count":84,"latest_published_at":85},"Dev Tools","dev-tools",84,"2026-09-26T04:20:58.000Z",{"name":87,"slug":88,"count":89,"latest_published_at":90},"Startups","startups",76,"2026-09-25T18:33:59.000Z",{"name":92,"slug":93,"count":94,"latest_published_at":95},"Gaming","gaming",48,"2026-09-25T18:35:21.000Z",{"name":97,"slug":98,"count":94,"latest_published_at":99},"General","general","2026-09-26T17:02:42.000Z",{"name":101,"slug":102,"count":103,"latest_published_at":104},"Reviews","reviews",30,"2026-09-24T20:07:31.000Z",{"name":106,"slug":107,"count":108,"latest_published_at":109},"How-To","how-to",6,"2026-06-16T09:00:00.000Z"]