A research team has published CHERRY, a three-part method for training smaller language models without proportionally smaller results.
The core trick is depth compression: a 48-layer, 1B-parameter transformer gets collapsed to 6 layers (227M parameters) by averaging adjacent layers, then restored through learned recurrent unrolling. With 34 effective recurrent layers, the compressed model reaches a held-out loss of 2.934 — close to, but not quite matching, a 566M dense model at 2.926, representing a 2.5x parameter reduction with a narrow gap in quality. Stacking several of these compressed models as a Mixture of Efficient Experts further closes the gap: a 2-expert setup hits 2.789, beating any single compressed model at comparable active parameters. A third technique, Selective Ground Truth Token Training, concentrates supervision on roughly 15% of tokens carrying semantic payload and claims 4.5x per-supervised-token efficiency — though the authors note that effect depends on natural-language structure and collapses on shuffled text.
Efficiency research has grown into its own arms race as inference costs keep climbing, and CHERRY's recurrent recovery angle is a different bet than the pruning and quantization approaches most labs lean on. Fitting competitive performance into a 227M footprint matters most at the edge, where memory and power budgets are fixed.
The team validates everything on CHERRY-1.8B, a Korean foundation model, and is upfront that their evidence covers one model family on one language using loss-based metrics only — a candor that is rarer than it should be in this field.