A new small language model gets more mileage out of fewer weights by looping the same layers over and over instead of stacking new ones.
Researchers trained "Looped GPT-BERT," a 12.18-million-parameter model, for the BabyLM 2026 Strict-small challenge, which limits training data to mimic what a child hears growing up. The model uses just four physical layers, but runs data through them twelve times in a row, combining GPT-BERT's masked-token and causal language-modeling objectives with this depth-wise parameter sharing. Trained on a 7.48-million-word English corpus, it scored an Overall Average of 35.42 and an NLP Average of 48.48 on the BabyLM leaderboard. Against public BabyLM 10M Strict-small GPT-2 and GPT-BERT baselines, it matched performance on tasks like BLiMP and GLUE despite using far fewer parameters.
This matters because it pokes at the assumption that better language models always need more parameters. If looping four layers twelve times can hold its own against models with more distinct layers, that is a cheap way to squeeze extra capability out of constrained hardware or data-scarce settings. But the paper's own loop ablations show the trick has a ceiling: extra recurrence helps some linguistic tasks while hurting others, suggesting a handful of physical layers just cannot hold enough representational room for everything.
So this is not a free lunch, more a different way to spend the same compute budget. Looping reads as a training-time workaround for small models, not a blueprint for replacing bigger ones.