Offline imitation of a noisy AI teacher requires exponentially more data than online interaction — and researchers now have the proof.
A new paper from arXiv argues that supervised fine-tuning, the workhorse of language model training, hits a hard wall when the teacher is imperfect. Researchers built a formal model in which a learner trains from a noisy version of an expert policy but must ultimately match a clean expert's performance. Under that framing, offline imitation learning needs sample complexity that grows exponentially with the decision horizon — a brutal scaling cliff. On-policy distillation, where the learner interacts with the teacher during training rather than just replaying recorded trajectories, reduces that dependence to polynomial, which is the difference between tractable and not.
This matters because the noisy-teacher scenario is not hypothetical. Training language models to produce long chains of thought is exactly this problem: the demonstrations used for supervised fine-tuning are generated by models that are themselves imperfect, not some platonic clean oracle. If the theory holds, it gives a principled reason to prefer on-policy methods even when they are more expensive to run.
The catch is that the proposed on-policy algorithm trades statistical efficiency for horizon efficiency — it scales worse with the size of the policy class, which in practice means larger models pay a steeper price. Whether that tradeoff is net positive in real LLM training pipelines is left as future work, which is the kind of sentence that tends to age slowly in machine learning research.