A new paper identifies a specific inefficiency in how AI agents train themselves, and offers a fix for it.
On-policy self-distillation lets a language agent learn from a privileged version of itself that grades its own trajectories token by token, and most systems weight that supervision purely by how much they trust the teacher's judgment on each token. A new paper argues that's the wrong criterion: trust doesn't indicate whether reinforcing a token actually helps the policy's training objective, a gap it calls the trust-utility mismatch. Its fix, Influence Calibration for Self-Distillation (ICSD), instead measures how much each token's supervision would shift the policy's own reinforcement-learning gradient, then reweights accordingly with no extra computation. Tested on ALFWorld, WebShop, and Search-QA across model sizes from 1.5B to 7B parameters, ICSD beat trust-only allocation under two training algorithms, reaching 96.1% success on ALFWorld and a 93.1 WebShop score at 7B.
The more striking number is how much trust-based training was wasting: the analysis found 60.1% of teacher-endorsed signal was going to tokens that actively worked against the policy's goals, a figure ICSD cut to 37.8% while better aligning the training signal with the real RL gradient. That's a meaningful inefficiency for labs building agents that act over many steps, like browsing a shop or completing a search task, since self-distillation is one of the cheaper ways to supervise those long trajectories without a separate reward model for every token.
Still, the gains are demonstrated on relatively small models and text-based simulation benchmarks, not frontier-scale agents in the wild, so it's an efficiency fix worth watching rather than a proven one at scale.