A research team has published a training framework that forces language models to explain their own errors before trying again — and then makes those explanations stick.
Self-Review Reinforcement Learning, or SRRL, inserts an explicit self-review step into each reinforcement learning episode. When a model's first attempt fails, it produces a written diagnosis of what went wrong, then uses that diagnosis to generate a second attempt. The key twist: unlike inference-time reflection methods that stay outside the training loop, SRRL uses policy gradients to optimize the self-review itself, and a selective distillation step writes successful corrections back into the base model. A cross-episode memory bank stores useful self-reviews and replays them when similar tasks appear later in training. The team tested the framework on Qwen 3-4B and OLMo-3-7B against a standard RLVR baseline using the GRPO optimizer on the GSM8K math benchmark, and SRRL outperformed on both final reward and learning efficiency.
Reinforcement learning for language models has always struggled with sparse feedback — the model finishes a long chain of reasoning, gets a binary right-or-wrong signal, and has to guess which step caused the failure. SRRL is a structural answer to that problem: make the model articulate the failure mode before the environment moves on. That internalized critique, rather than an external verifier, becomes the training signal.
The comparison to Reflexion — a well-known inference-time reflection technique — is worth sitting with. Reflexion improves outputs at runtime but leaves the underlying model unchanged; SRRL claims to make the lesson permanent. Whether that claim holds beyond a math benchmark, and at scales larger than 7B parameters, is the question the paper leaves open.