Teaching a small AI model by grading its bigger 'teacher' models on correctness makes it noticeably more accurate. But a new study finds the trick quietly breaks the model's ability to flag false claims, and it still can't prove the model's answers are backed by real evidence.
Researchers ran a controlled distillation experiment: a 63.9-million-parameter student model was trained on outputs from teacher models, using 4,330 source documents, 12,990 training examples, and 406 optimization updates, then tested on 267 held-out examples across three random seeds. Eight training setups were compared, including one that weighted the teachers' answers by whether the underlying decision was actually correct. Against an unfiltered baseline, that correctness-weighted setup lifted accuracy by 16.6 percentage points, raised a five-category F1 score by 13.2 points, and cut a measured unsafe-action rate by roughly half. A simpler approach, just discarding wrong answers instead of weighting them, already reached 66% accuracy and a 0.53 F1 score on its own, and the more complex weighting scheme added nothing on top of it.
The costs show up in the details reviewers rarely publish. Across every test run, the weighted model never once correctly identified a claim as 'Refuted', it simply stopped calling anything false. In two of three runs, it labeled all 167 fact-check examples as 'not enough info', a blanket non-answer. A follow-up audit checking whether outputs were actually backed by cited evidence found the weighted model did worse than the unfiltered one, 0 of 20 supported answers versus 1 of 20, though the researchers flag their own audit as too loosely controlled to call that a real result.
The finding is a reminder that a training signal aimed at one number can quietly reshape everything else a model does. Here, the closest thing to a fact-check on the method's own honesty found it still hasn't proven the thing it set out to prove.