A student neural network trained on pure random noise can still absorb its teacher's classification ability, as long as both models share the same initial weights.
Researchers working in an MLP distillation setup on MNIST found that hidden-channel transfer between teacher and student models depends on geometry, not information content. Shared initialization makes the output projection W_2 a common coordinate key. KL divergence gradients then reshape the student's input projection W_0 until the student's hidden representations align with the teacher's. They called this "covert trait propagation" (CTP) and validated it with five experiments, including the finding that multi-teacher ensembles cancel each other out despite each teacher carrying comparable label information, and a linear centered kernel alignment score of r=0.98 between representation alignment and student accuracy across a continuous initialization sweep.
The freeze test is the clearest proof: locking W_0 destroys transfer entirely, while locking W_2 leaves it intact, confirming that W_2's shared geometry gates the channel while W_0 is where learning actually happens. The paper also applies this lens to cross-token behavioral entanglement in instruction-tuned LLMs, finding that a standard log-ratio metric used to measure it produces an apparent frequency bias that is largely a circularity artifact. That puts prior alignment measurements relying on that metric on shakier ground.
The practical implication is that fine-tuning from a shared pretrained checkpoint is not a clean slate: initialization coordinates travel with the weights, and so may whatever traits the original model carried.