A new theoretical paper argues that a popular fix for AI models forgetting old tasks has been protecting the wrong layers all along.
Continual learning regularizers like EWC try to stop a model from forgetting earlier tasks by penalizing changes to parameters that mattered before, scoring each parameter with something called diagonal Fisher information. The researchers show that under a block-diagonal assumption about the network's Hessian, forgetting actually splits into per-layer terms driven by each layer's top eigenvalue, a quantity the diagonal Fisher score cannot capture. Two layers can share the same average Fisher value and still have top eigenvalues that differ by a factor as large as the layer's width. Spreading regularization evenly across parameters, rather than by layer, costs new-task performance in proportion to how ill-conditioned that layer is.
The proposed fix is almost embarrassingly simple: protect early layers heavily and let deeper layers adapt more freely, instead of treating every parameter as equally important. Applied to EWC and the newer SLCA method, this layer-adaptive approach produced measurable gains in both average performance and forgetting metrics.
It is a useful correction to an assumption baked into a decade of continual learning work: that scoring importance per-parameter is automatically more precise than scoring it per-layer. Sometimes the coarser unit is where the real signal lives.