[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"branding":3,"analytics":7,"article-a-noise-trained-model-can-still-inherit-its-teachers-skills":10,"sections":41},{"siteName":4,"siteTagline":5,"publisherName":4,"contactEmail":6},"The Revision","Tech news, decoded.","editor@therevision.news",{"gaMeasurementId":8,"adsenseClientId":9},"G-ZW2MV82GYR","ca-pub-8533917693782264",{"article":11},{"id":12,"slug":13,"title":14,"dek":15,"body_md":16,"tags_json":17,"published_at":18,"created_at":19,"updated_at":20,"status":21,"review_note":22,"review_notes":23,"image_url":22,"persona_id":22,"persona_name":22,"section":30,"tags":31,"sources":36,"feedback":40,"feedback_at":22,"cost_usd":40,"total_tokens":40},3943,"a-noise-trained-model-can-still-inherit-its-teachers-skills","A Noise-Trained Model Can Still Inherit Its Teacher's Skills","Shared weight initialization creates a geometric channel that lets student models absorb teacher skills, even when trained on pure random noise.","A student neural network trained on pure random noise can still absorb its teacher's classification ability, as long as both models share the same initial weights.\n\nResearchers working in an MLP distillation setup on MNIST found that hidden-channel transfer between teacher and student models depends on geometry, not information content. Shared initialization makes the output projection W_2 a common coordinate key. KL divergence gradients then reshape the student's input projection W_0 until the student's hidden representations align with the teacher's. They called this \"covert trait propagation\" (CTP) and validated it with five experiments, including the finding that multi-teacher ensembles cancel each other out despite each teacher carrying comparable label information, and a linear centered kernel alignment score of r=0.98 between representation alignment and student accuracy across a continuous initialization sweep.\n\nThe freeze test is the clearest proof: locking W_0 destroys transfer entirely, while locking W_2 leaves it intact, confirming that W_2's shared geometry gates the channel while W_0 is where learning actually happens. The paper also applies this lens to cross-token behavioral entanglement in instruction-tuned LLMs, finding that a standard log-ratio metric used to measure it produces an apparent frequency bias that is largely a circularity artifact. That puts prior alignment measurements relying on that metric on shakier ground.\n\nThe practical implication is that fine-tuning from a shared pretrained checkpoint is not a clean slate: initialization coordinates travel with the weights, and so may whatever traits the original model carried.","[\"knowledge distillation\",\"machine learning\",\"model alignment\",\"llm\"]","2026-07-07T04:00:00.000Z","2026-07-07T12:50:44.799Z","2026-07-07T12:50:47.652Z","published",null,[24],{"id":25,"reviewer":26,"round":27,"reason":28,"status":29},"editor-r1","editor",1,"The article misstates the freezing result: the source says freezing W_2 (the output projection) leaves transfer intact and freezing W_0 (the input projection) destroys it, but the draft inverts these, calling W_0 the 'input layer' and W_2 the 'output layer' without the variable names — which is accurate labeling — yet then says 'freezing the student's input layer kills transfer entirely while freezing its output layer leaves transfer intact,' which matches the source; however the draft also omit","resolved","ai",[32,33,34,35],"knowledge distillation","machine learning","model alignment","llm",[37],{"name":38,"url":39},"arXiv cs.AI","https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.04432",0,{"sections":42},[43,47,52,57,62,67,72,77,82,86,91,95,100,105],{"name":44,"slug":30,"count":45,"latest_published_at":46},"AI",2590,"2026-07-16T04:00:00.000Z",{"name":48,"slug":49,"count":50,"latest_published_at":51},"Security","security",294,"2026-07-15T19:59:48.000Z",{"name":53,"slug":54,"count":55,"latest_published_at":56},"Deals","deals",179,"2026-06-29T20:02:07.000Z",{"name":58,"slug":59,"count":60,"latest_published_at":61},"Policy","policy",158,"2026-07-16T00:02:48.000Z",{"name":63,"slug":64,"count":65,"latest_published_at":66},"Hardware","hardware",122,"2026-07-14T19:46:26.000Z",{"name":68,"slug":69,"count":70,"latest_published_at":71},"Consumer Tech","consumer-tech",93,"2026-07-13T13:20:48.000Z",{"name":73,"slug":74,"count":75,"latest_published_at":76},"Software","software",70,"2026-07-13T19:52:25.000Z",{"name":78,"slug":79,"count":80,"latest_published_at":81},"Science","science",66,"2026-07-10T10:29:37.000Z",{"name":83,"slug":84,"count":85,"latest_published_at":18},"Dev Tools","dev-tools",59,{"name":87,"slug":88,"count":89,"latest_published_at":90},"Gaming","gaming",41,"2026-07-09T04:00:00.000Z",{"name":92,"slug":93,"count":89,"latest_published_at":94},"Startups","startups","2026-06-29T20:55:50.000Z",{"name":96,"slug":97,"count":98,"latest_published_at":99},"General","general",29,"2026-07-10T22:28:58.000Z",{"name":101,"slug":102,"count":103,"latest_published_at":104},"Reviews","reviews",20,"2026-06-24T12:00:01.000Z",{"name":106,"slug":107,"count":108,"latest_published_at":109},"How-To","how-to",6,"2026-06-16T09:00:00.000Z"]