[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"branding":3,"analytics":7,"article-new-method-lets-ai-testing-platforms-add-and-cut-treatments-safely":10,"sections":34},{"siteName":4,"siteTagline":5,"publisherName":4,"contactEmail":6},"The Revision","Tech news, decoded.","editor@therevision.news",{"gaMeasurementId":8,"adsenseClientId":9},"G-ZW2MV82GYR","ca-pub-8533917693782264",{"article":11},{"id":12,"slug":13,"title":14,"dek":15,"body_md":16,"tags_json":17,"published_at":18,"created_at":19,"updated_at":20,"status":21,"review_note":22,"review_notes":23,"image_url":22,"persona_id":22,"persona_name":22,"section":24,"tags":25,"sources":29,"feedback":33,"feedback_at":22,"cost_usd":33,"total_tokens":33},9053,"new-method-lets-ai-testing-platforms-add-and-cut-treatments-safely","New Method Lets AI Testing Platforms Add and Cut Treatments Safely","A new statistical framework lets AI-driven experimentation platforms add and drop treatments on the fly while still controlling false discoveries.","A team of researchers has worked out the math for running AI experiments that never stop.\n\nGenerative AI has made it trivial to spin up new treatments faster than old-school experimentation platforms can handle, whether that is a new drug formulation or a new ad creative. The researchers formalize this \"always-on\" setting, where treatments are added to and pulled from a live experiment on a rolling basis, and build sequential statistical tests that decide whether to accept or reject each one while controlling the false discovery rate even as the schedule keeps shifting. Their method, built on a \"testing-by-betting\" framework, constructs test supermartingales for each treatment's average effect and holds up under arbitrary stopping times. The authors show the approach is growth-rate optimal in an almost-sure sense, meaning it extracts statistical evidence about as efficiently as theory allows.\n\nClassic A\u002FB testing assumes a fixed set of treatments with a clean start and end date, assumptions that break when a platform can generate and discard variants continuously, as marketing and recommendation systems increasingly do. Without time-uniform error control, continuously adding tests quietly inflates false discoveries, which is exactly the kind of statistical sloppiness that makes \"data-driven\" claims hollow. This work gives platforms a rigorous way to keep experimenting indefinitely without the false-positive rate creeping up unnoticed.\n\nThe bottleneck in AI-generated experimentation was never generating ideas. It was knowing, honestly, which ones actually worked.","[\"ai\",\"statistics\",\"experimentation\",\"research\"]","2026-10-01T04:00:00.000Z","2026-10-01T17:33:12.144Z","2026-10-01T17:33:18.097Z","published",null,[],"ai",[24,26,27,28],"statistics","experimentation","research",[30],{"name":31,"url":32},"arXiv cs.AI","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.38695",0,{"sections":35},[36,39,43,48,53,58,62,67,72,76,81,86,91,96],{"name":37,"slug":24,"count":38,"latest_published_at":18},"AI",5488,{"name":40,"slug":41,"count":42,"latest_published_at":18},"Security","security",809,{"name":44,"slug":45,"count":46,"latest_published_at":47},"Policy","policy",429,"2026-10-01T02:26:17.000Z",{"name":49,"slug":50,"count":51,"latest_published_at":52},"Deals","deals",298,"2026-09-30T21:00:26.000Z",{"name":54,"slug":55,"count":56,"latest_published_at":57},"Hardware","hardware",196,"2026-09-30T13:00:00.000Z",{"name":59,"slug":60,"count":61,"latest_published_at":18},"Science","science",162,{"name":63,"slug":64,"count":65,"latest_published_at":66},"Consumer Tech","consumer-tech",149,"2026-09-30T22:57:11.000Z",{"name":68,"slug":69,"count":70,"latest_published_at":71},"Dev Tools","dev-tools",93,"2026-10-01T02:30:48.000Z",{"name":73,"slug":74,"count":70,"latest_published_at":75},"Software","software","2026-09-30T21:41:11.000Z",{"name":77,"slug":78,"count":79,"latest_published_at":80},"Startups","startups",84,"2026-09-30T20:39:09.000Z",{"name":82,"slug":83,"count":84,"latest_published_at":85},"Gaming","gaming",51,"2026-09-30T16:24:30.000Z",{"name":87,"slug":88,"count":89,"latest_published_at":90},"General","general",50,"2026-09-30T21:37:54.000Z",{"name":92,"slug":93,"count":94,"latest_published_at":95},"Reviews","reviews",31,"2026-09-28T14:31:34.000Z",{"name":97,"slug":98,"count":99,"latest_published_at":100},"How-To","how-to",6,"2026-06-16T09:00:00.000Z"]