A new paper puts an exact number on the memory loss models suffer when they learn tasks one after another instead of all at once.
The researchers modeled continual learning with overparameterized linear regression, feeding a model one task at a time instead of all at once. They proved that forgetting and overall error converge to the same long-run number, which splits into the error joint training would produce plus an extra sequential price for learning in order; in similarly-shaped tasks, that price doubles the total error versus training on everything together. They then tested elastic weight consolidation, a technique that penalizes a model for drifting from what earlier tasks taught it, and found a direct tradeoff: cranking up the penalty shrinks the sequential price but slows convergence by the same factor. Running the formula against real Jester joke ratings, where users disagree constantly, it correctly predicted both the cost of those conflicting tastes and how much the technique clawed back.
Continual learning is the approach behind any system that needs to keep absorbing new information without being retrained from scratch, recommendation engines, robots, personalized assistants. Most work on the field's central problem, catastrophic forgetting, has been empirical: try a fix, see if the numbers improve. This paper gives that problem an actual price tag and shows that a standard fix does not eliminate the cost, it just trades it for slower learning.
The math is done on linear regression, not the deep networks powering today's chatbots, so treat the exact multiplier with caution, but a field long light on hard guarantees just got one.