A new paper proposes a unified explanation for why knowledge distillation works - and a technique to make it work better.
Knowledge distillation is how the AI field shrinks large language models into smaller, cheaper ones: the big "teacher" model trains a smaller "student" to mimic its outputs. Researchers from multiple institutions decomposed LLM output scores into what they call interactions - nonlinear relationships between input words. They found that across all major distillation methods, the student model survives by becoming sparser: it drops most interactions and keeps only the ones it needs. The variance in quality between different distillation approaches, they argue, comes down to how well each method handles complex, multi-word interactions.
That framing matters because it turns a black-box process into something measurable. Instead of tuning distillation by trial and error, developers could target interaction sparsity directly - and the paper backs that up with a plug-and-play loss function called Complex Interaction Penalty (CIP) that consistently improved performance across both in-domain and out-of-distribution benchmarks.
Knowledge distillation has been a workhorse technique for years - it underlies many of the small, fast models that run inference cheaply at scale - but the field has mostly treated it as empirically validated without a strong theoretical account. If the interaction-sparsity framing holds up under scrutiny, it could give practitioners a cleaner knob to turn rather than a menu of methods to guess from.