A new paper describes a way to keep small AI models learning new tricks without erasing the old ones.
Researchers introduce SynthSelf, a continual post-training method built for small models used in low-latency, on-device settings. It builds on an earlier technique called Self-Distillation Fine-Tuning, which generates training data from a model's own in-context demonstrations, but that approach falls apart on small models that lack strong in-context learning and instruction-following skills. The team traced the failure to distribution shift: the more a model's self-generated training data strays from its original distribution, the more it forgets old skills while learning new ones. SynthSelf fixes this by handing the rewriting step to a slightly larger model from the same family, then filtering the output with off-policy supervision and rejection sampling, so the small model gets high-quality training data that stays close to its own distribution.
Across four acquisition tasks and fourteen retention benchmarks, SynthSelf beat six rival methods, including both on-policy and off-policy distillation, at three different small model sizes. That's notable because most continual-learning research assumes a model big enough to lean on its own in-context tricks - a mismatch for the compact models actually running on phones and laptops.
It's a narrow fix, but a useful one: the same-family, slightly-bigger-teacher trick swaps a small model's self-confidence for someone else's judgment, without asking it to outgrow its own distribution.