A new training method teaches AI models to fix their own mistakes, no answer key required.
Researchers describe ReTeach, a self-distillation framework, in an arXiv paper published October 9, 2026, that trains a model to become its own teacher using only its own failed attempts. Starting from a wrong answer, the system alternates between reflecting on the mistake and retrying, repeating the cycle until it succeeds or runs out of retries - with no reference solutions, external feedback, or memory of other examples involved. It sorts the results into three buckets - answers right on the first try, answers fixed through reflection, and answers that stayed wrong - weighting each differently when training the student to match the teacher's corrected predictions through single-pass, on-policy distillation. Across six benchmarks spanning math, science question answering, and tool use, ReTeach beat the GRPO reinforcement-learning baseline by 1.39 percentage points on average.
Most self-distillation setups need either a stronger teacher model, a labeled answer key, or rich feedback signals - resources that are often missing when fine-tuning outside a lab. ReTeach manufactures that advantage from nothing but a model's own trial and error, which matters for anyone fine-tuning on tasks where ground-truth answers are scarce or expensive to collect. A 1.39-point average gain over GRPO is real but modest - useful, not transformative.
Self-taught models can only learn from mistakes they eventually correct within their retry budget, so a problem a model can never solve stays a blind spot no amount of reflection will fix.