[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"branding":3,"analytics":7,"article-researchers-find-which-transformer-parameters-training-cant-see":10,"sections":41},{"siteName":4,"siteTagline":5,"publisherName":4,"contactEmail":6},"The Revision","Tech news, decoded.","editor@therevision.news",{"gaMeasurementId":8,"adsenseClientId":9},"G-ZW2MV82GYR","ca-pub-8533917693782264",{"article":11},{"id":12,"slug":13,"title":14,"dek":15,"body_md":16,"tags_json":17,"published_at":18,"created_at":19,"updated_at":20,"status":21,"review_note":22,"review_notes":23,"image_url":22,"persona_id":22,"persona_name":22,"section":30,"tags":31,"sources":36,"feedback":40,"feedback_at":22,"cost_usd":40,"total_tokens":40},4243,"researchers-find-which-transformer-parameters-training-cant-see","Researchers Find Which Transformer Parameters Training Can't See","Treating Transformer pretraining as a fast-slow ODE shows a computable depth past which decaying-bundle weight changes are invisible to training.","New research treats Transformer pretraining as a dynamical system — and draws a line through parameter space between weights that training can see and weights it cannot.\n\nA paper posted this week recasts Transformer pretraining as a fast-slow singularly perturbed flow, where depth plays the role of time. The key result: past a computable saturation depth, weight perturbations that live on the \"decaying bundle\" — the mathematically contracting directions in the linearized dynamics — produce no first-order change in either the model's trajectory or its frozen attention kernel. The paper labels these perturbations \"invisible\" and shows that the cross-block couplings that actually drive prediction sit on the \"visible\" side instead. Whether a cross-block channel helps at all depends on data: if neighboring blocks share no structure, even visible weight adjustments contribute nothing to prediction risk.\n\nResearchers have long optimized Transformers without a clear map of which parameters actually move the needle; this analysis offers the first principled partition of parameter space on formal dynamical grounds. If the invisible directions are large, gradient-based training is, in an exact technical sense, spending compute on perturbations that cannot affect the loss — a concrete handle on longstanding puzzles around training efficiency and overparameterization.\n\nThe framework does not yet come with a recipe for pruning invisible parameters during training, but it offers something rarer: a formal bound on what a Transformer is mathematically capable of learning, given a stability margin and the structure of its data.","[\"transformers\",\"machine learning\",\"ai research\",\"neural networks\"]","2026-07-07T04:00:00.000Z","2026-07-07T21:01:43.787Z","2026-07-07T21:01:46.600Z","published",null,[24],{"id":25,"reviewer":26,"round":27,"reason":28,"status":29},"editor-r1","editor",1,"The headline and dek are vague placeholders — the headline buries the actual finding ('fast-slow dynamical system,' 'visible\u002Finvisible parameter directions') behind a generic 'math framework explains it' framing, and the dek uses soft, imprecise language ('ones that matter and ones that don't') instead of the concrete claim from the source (parameter perturbations on the decaying bundle are invisible to training signal past a computable saturation depth); rewrite both to state the specific resul","resolved","ai",[32,33,34,35],"transformers","machine learning","ai research","neural networks",[37],{"name":38,"url":39},"arXiv cs.AI","https:\u002F\u002Farxiv.org\u002Fabs\u002F2606.16730",0,{"sections":42},[43,47,52,57,62,67,72,77,82,86,91,95,100,105],{"name":44,"slug":30,"count":45,"latest_published_at":46},"AI",2590,"2026-07-16T04:00:00.000Z",{"name":48,"slug":49,"count":50,"latest_published_at":51},"Security","security",294,"2026-07-15T19:59:48.000Z",{"name":53,"slug":54,"count":55,"latest_published_at":56},"Deals","deals",179,"2026-06-29T20:02:07.000Z",{"name":58,"slug":59,"count":60,"latest_published_at":61},"Policy","policy",158,"2026-07-16T00:02:48.000Z",{"name":63,"slug":64,"count":65,"latest_published_at":66},"Hardware","hardware",122,"2026-07-14T19:46:26.000Z",{"name":68,"slug":69,"count":70,"latest_published_at":71},"Consumer Tech","consumer-tech",93,"2026-07-13T13:20:48.000Z",{"name":73,"slug":74,"count":75,"latest_published_at":76},"Software","software",70,"2026-07-13T19:52:25.000Z",{"name":78,"slug":79,"count":80,"latest_published_at":81},"Science","science",66,"2026-07-10T10:29:37.000Z",{"name":83,"slug":84,"count":85,"latest_published_at":18},"Dev Tools","dev-tools",59,{"name":87,"slug":88,"count":89,"latest_published_at":90},"Gaming","gaming",41,"2026-07-09T04:00:00.000Z",{"name":92,"slug":93,"count":89,"latest_published_at":94},"Startups","startups","2026-06-29T20:55:50.000Z",{"name":96,"slug":97,"count":98,"latest_published_at":99},"General","general",29,"2026-07-10T22:28:58.000Z",{"name":101,"slug":102,"count":103,"latest_published_at":104},"Reviews","reviews",20,"2026-06-24T12:00:01.000Z",{"name":106,"slug":107,"count":108,"latest_published_at":109},"How-To","how-to",6,"2026-06-16T09:00:00.000Z"]