Researchers have built a fine-tuning method that lets AI models decide, token by token, how much of a teacher model's expertise to absorb.
The technique, called iSDFT, tweaks a training method known as self-distillation fine-tuning, where a model learns new skills by studying a stronger "teacher" version of itself performing a task. The catch with the standard approach: it always aims for the teacher's full answer, with no dial for how much influence that teacher should have at each step. iSDFT adds that dial. At every token it picks the closest possible answer to what the student already believes while staying within a set budget for teacher information, and it anchors the model to its original, unfine-tuned self to limit drift over time.
That distinction matters because most fixes for catastrophic forgetting, the tendency of a model to lose old skills while learning new ones, treat the symptom rather than the training signal itself. Metering the teacher's influence is a more surgical approach than freezing layers or bolting on regularization penalties after the fact. Tested on four language models across two specialization tasks, the method beat standard self-distillation in seven of eight settings, stayed within half a point of baseline performance 73% of the time versus 52% for the strongest prior approach, and posted the largest average gains across ten separate math and coding benchmarks.
Those numbers come from four model families and two tasks chosen by the authors, so it is still an open question whether the dial works as cleanly once it meets larger, messier production models.