AI models can look fine on paper while quietly forgetting what made them good.
A new paper introduces VISTA, short for Validation-Informed Trajectory Adaptation via Self-Distillation, an online training framework built to catch a failure mode the researchers call Trajectory Deviation. As training goes on, a model can abandon high-generalization behavior for specific slices of its data and discard previously learned features, all without tripping the usual overfitting warning signs. VISTA scores earlier checkpoints, which it calls expert anchors, using a validation-informed Marginal Coverage metric that finds states still holding specialized competence over particular data regions. It then blends a coverage-weighted ensemble of those anchors into training online, regularizing the loss landscape so mastered knowledge sticks around, and a lightweight version of the setup cuts storage overhead by 90 percent with no performance loss, according to the paper's benchmarks.
The standard practice of watching validation accuracy and stopping when it plateaus assumes a rising validation curve means the model is improving everywhere in the data. Trajectory Deviation is a reminder that one aggregate number can hide regression on specific subgroups, which matters more once models leave clean benchmark splits and hit messy real-world data. VISTA is also a cheaper cousin of standard self-distillation and ensembling, which usually require storing full model copies rather than a compressed summary of a few checkpoints.
The 90 percent storage savings and robustness gains come from the authors' own benchmarks, so like most single-paper results in the self-distillation literature, they still need independent replication before anyone rewrites their training pipeline around them.