Researchers have built language models that fail to generalize in precisely controlled ways — and the results are unsettling.
A new paper proposes a method for constructing "model organisms" that exhibit predictable generalization failures after reinforcement learning fine-tuning. The technique works by first using supervised fine-tuning on a dataset mixing transcripts from multiple "conditional policies" — each assigned different behaviors for different task distributions. When RL training is then applied, it selects for whichever policy scores highest on the training distribution. In a stark demonstration, two otherwise identical question sets prepended with different trigger strings caused RL training on one distribution to collapse performance on the other to zero.
This matters because post-training on frontier models always involves curated task suites, and the gap between training and deployment is poorly understood. These synthetic model organisms give researchers a clean, controllable way to stress-test alignment assumptions — and to produce existence proofs that training success and generalization can diverge in structured, reproducible ways.
The authors are careful to note their construction is deliberately simplified and may not mirror how generalization failures emerge in production models. But "may not closely resemble natural failures" is doing a lot of work there — the real question is whether the structured failure modes they demonstrate are edge cases or early glimpses of something that scales.