A new fine-tuning method tries to stop large language models from forgetting what they already knew.
The technique, called EoupCT, targets catastrophic forgetting: the tendency of a model to lose old skills or general knowledge when it gets fine-tuned on a new task. Existing fixes like orthogonal gradient projection help models retain skills across a sequence of fine-tuning jobs, but they need the original pre-training data and gradients, which are usually unavailable for an off-the-shelf model. EoupCT works around that gap by generating synthetic "pseudo data" designed to resemble whatever the model is most likely to forget, using a soft prompt tuned with a method called Gumbel-Softmax relaxation. It then estimates what the original pre-training gradients would look like from that data and mathematically forces new training updates to move at right angles to them, aiming to preserve old knowledge while the model learns the new task.
Catastrophic forgetting is one of the quiet costs of continual fine-tuning: a model improves at your specific job and gets worse at everything else, and most teams fine-tuning a model don't have the original training data on hand to check for that decay. If a technique like this holds up outside the paper's own tests, it could let companies update deployed models repeatedly without re-running a full battery of general-capability checks each time.
That's a real problem for anyone fine-tuning a closed model they didn't train themselves. As with most forgetting-mitigation papers, the real test will be whether it survives messier, real-world fine-tuning pipelines rather than curated benchmarks.