A large language model can know that Alice's father is Bob but fail to answer who Bob's daughter is, and a new paper fixes that gap before it spreads to smaller models.
Researchers behind the paper call this a directional limitation: a teacher model may recall a parent-child relation in one direction but not the other, even though it can still recognize a correct answer when scored in the direction it actually knows. Their method, directional label distillation, has the frozen teacher score candidate answers in that known direction rather than generate an answer in the direction being asked. The best-scoring candidate becomes the training label for the smaller student model. Tested on facts about parents and children, this known-direction scoring produced more accurate labels than scoring the requested direction, even after researchers corrected for name-frequency bias.
Standard distillation just copies whatever the teacher generates, so a teacher's blind spot becomes the student's blind spot too. In this study, students trained on known-direction labels scored 13 to 15 points higher on open-ended accuracy than students trained on direction-corrected reverse labels, with gains holding even on unscreened queries. That is a meaningful fix for an industry racing to package knowledge from huge models into smaller, cheaper ones.
It will not make a small model omniscient, but it is a reminder that model knowledge has a grain, and ignoring it just teaches the next model the same blind spots.