[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"branding":3,"analytics":7,"article-researchers-chart-the-risks-of-ai-that-improves-itself":10,"sections":35},{"siteName":4,"siteTagline":5,"publisherName":4,"contactEmail":6},"The Revision","Tech news, decoded.","editor@therevision.news",{"gaMeasurementId":8,"adsenseClientId":9},"G-ZW2MV82GYR","ca-pub-8533917693782264",{"article":11},{"id":12,"slug":13,"title":14,"dek":15,"body_md":16,"tags_json":17,"published_at":18,"created_at":19,"updated_at":20,"status":21,"review_note":22,"review_notes":23,"image_url":22,"persona_id":22,"persona_name":22,"section":24,"tags":25,"sources":30,"feedback":34,"feedback_at":22,"cost_usd":34,"total_tokens":34},8217,"researchers-chart-the-risks-of-ai-that-improves-itself","Researchers Chart the Risks of AI That Improves Itself","A new paper treats AI safety as an evolving problem, tracking drift and eroding guarantees as systems improve themselves over generations.","Researchers have a new name for an old fear: AI systems that get worse at staying safe the longer they keep improving themselves.\n\nA team publishing on arXiv introduces evolutionary safety, a framework for studying AI that trains on its own experience, updates its own models, and produces new versions of itself over time, a process called recursive self-improvement. The paper does not test a specific model. Instead it catalogs how safety properties can quietly degrade across generations, naming failure modes like intent drift, error accumulation, experience contamination, and evaluator drift, where the system used to judge safety becomes compromised itself. It sorts these risks across five areas: an agent's persistent state, its model weights, its evaluation and environment feedback, the computing substrate it runs on, and the meta-level processes that generate its successors. From that map, the authors propose governance rules for how changes get made, selected, authorized, tracked, and reversed.\n\nMost AI safety testing still treats a model as a fixed thing you check once and ship. This paper argues that approach collapses once a system starts training on its own output or spawning updated copies of itself, because a clean safety check today says nothing about generation ten. That gap matters well beyond research labs, given how much of the current AI agent hype is built on systems that adapt and update themselves in production.\n\nThe paper is honest that it is proposing a taxonomy, not a solved problem. The evaluation tools it points to are still described as in progress, which is a fair summary of where AI safety research sits relative to how fast these systems are being deployed.","[\"ai safety\",\"recursive self-improvement\",\"ai research\",\"machine learning\"]","2026-09-28T04:00:00.000Z","2026-09-28T17:19:52.151Z","2026-09-28T17:19:59.145Z","published",null,[],"ai",[26,27,28,29],"ai safety","recursive self-improvement","ai research","machine learning",[31],{"name":32,"url":33},"arXiv cs.AI","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.31186",0,{"sections":36},[37,40,44,49,54,59,63,68,73,78,83,88,92,97],{"name":38,"slug":24,"count":39,"latest_published_at":18},"AI",4844,{"name":41,"slug":42,"count":43,"latest_published_at":18},"Security","security",762,{"name":45,"slug":46,"count":47,"latest_published_at":48},"Policy","policy",399,"2026-09-27T18:39:02.000Z",{"name":50,"slug":51,"count":52,"latest_published_at":53},"Deals","deals",266,"2026-09-28T14:00:00.000Z",{"name":55,"slug":56,"count":57,"latest_published_at":58},"Hardware","hardware",189,"2026-09-28T10:52:40.000Z",{"name":60,"slug":61,"count":62,"latest_published_at":18},"Science","science",151,{"name":64,"slug":65,"count":66,"latest_published_at":67},"Consumer Tech","consumer-tech",135,"2026-09-26T14:30:00.000Z",{"name":69,"slug":70,"count":71,"latest_published_at":72},"Software","software",91,"2026-09-25T20:55:00.000Z",{"name":74,"slug":75,"count":76,"latest_published_at":77},"Dev Tools","dev-tools",84,"2026-09-26T04:20:58.000Z",{"name":79,"slug":80,"count":81,"latest_published_at":82},"Startups","startups",76,"2026-09-25T18:33:59.000Z",{"name":84,"slug":85,"count":86,"latest_published_at":87},"Gaming","gaming",48,"2026-09-25T18:35:21.000Z",{"name":89,"slug":90,"count":86,"latest_published_at":91},"General","general","2026-09-26T17:02:42.000Z",{"name":93,"slug":94,"count":95,"latest_published_at":96},"Reviews","reviews",30,"2026-09-24T20:07:31.000Z",{"name":98,"slug":99,"count":100,"latest_published_at":101},"How-To","how-to",6,"2026-06-16T09:00:00.000Z"]