A new curriculum-based training trick promises to shrink large language models in half the time and memory.
Researchers propose layer-wise curriculum learning, a method for compressing large language models by having a smaller student model learn from a larger teacher model in stages. The model is split into segments of layers, and training starts with easier segments before moving to harder ones, rather than optimizing the whole network at once. The paper argues this ordering counters what it calls cumulative error, where small mistakes in early layers pile up and destabilize the rest of training. To keep the process fast, the team also built a feature-caching system with multi-threading that keeps layer outputs aligned and GPU utilization high.
On BERT and GPT-2, the approach cut GPU memory use and training time by more than 50 percent versus existing methods. It also outperformed other pruning techniques on LLaMA-family and Qwen models while using less memory in the same training time, suggesting the gains hold as models scale.
Compression research rarely makes headlines, but it determines which labs and startups can afford to run these models at all. A technique that halves training cost deserves more attention than another benchmark-topping model release.