A new training technique aims to stop AI language models from forgetting earlier skills when they learn new ones.
Researchers studied LoRA (low-rank adaptation), a popular way to teach large language models new tasks without retraining the whole system. A technique called O-LoRA tried to stop models from forgetting old tasks by forcing each new task's parameters to point in mathematically distinct directions, a constraint known as orthogonal projection. The new paper argues that constraint backfires: it blocks models from sharing useful knowledge between related tasks, a tradeoff the authors dub the "orthogonality dilemma." Their proposed fix, CoDe-LoRA, splits learning into two tracks, one that consolidates knowledge shared across tasks and one that keeps task-specific details separate, using a routing mechanism to decide which track handles what. Tested across four base models and three benchmark suites, CoDe-LoRA posted the highest average accuracy among the methods compared, without needing to replay old training data.
Catastrophic forgetting is the reason most AI systems get retrained from scratch instead of updated incrementally. That matters for any product that ships new features to a deployed model: replay-free continual learning, if it holds up, could make incremental updates cheaper and less risky than full retraining cycles.
The accuracy gains are self-reported on benchmarks the authors picked, so treat "best average accuracy" as a lab result, not proof it survives contact with a live product.