New research treats Transformer pretraining as a dynamical system — and draws a line through parameter space between weights that training can see and weights it cannot.
A paper posted this week recasts Transformer pretraining as a fast-slow singularly perturbed flow, where depth plays the role of time. The key result: past a computable saturation depth, weight perturbations that live on the "decaying bundle" — the mathematically contracting directions in the linearized dynamics — produce no first-order change in either the model's trajectory or its frozen attention kernel. The paper labels these perturbations "invisible" and shows that the cross-block couplings that actually drive prediction sit on the "visible" side instead. Whether a cross-block channel helps at all depends on data: if neighboring blocks share no structure, even visible weight adjustments contribute nothing to prediction risk.
Researchers have long optimized Transformers without a clear map of which parameters actually move the needle; this analysis offers the first principled partition of parameter space on formal dynamical grounds. If the invisible directions are large, gradient-based training is, in an exact technical sense, spending compute on perturbations that cannot affect the loss — a concrete handle on longstanding puzzles around training efficiency and overparameterization.
The framework does not yet come with a recipe for pruning invisible parameters during training, but it offers something rarer: a formal bound on what a Transformer is mathematically capable of learning, given a stability margin and the structure of its data.