A new academic benchmark finds that today's favorite pretraining trick for time-series models doesn't work nearly as well as advertised once you control for parameters and data.
Researchers tested seven self-supervised learning (SSL) methods across five major pretraining paradigms, pitting them against models trained from scratch on anomaly detection, classification, and forecasting tasks. They matched parameter counts and data budgets so the comparison wasn't skewed by bigger pretrained models simply having more capacity to work with. The results split sharply by task: SSL produced clear gains in anomaly detection and gave classification models a useful head start when fine-tuned. Forecasting was the outlier - pretraining offered little to no edge over models that never saw a pretext task at all.
The finding undercuts a common assumption, borrowed from vision and language research, that self-supervised pretraining is a universal lift for any kind of data. It also means linear probing - the cheap shortcut researchers often use to estimate how a pretrained model will perform - didn't reliably predict how those same models did after full fine-tuning, so teams relying on it to save compute may be fooling themselves. On top of that, synthetic training data held up well against real-world corpora, and making encoders deeper actively hurt forecasting accuracy instead of helping it.
Pretraining is not a free lunch across every kind of data, and anyone bolting SSL onto a forecasting pipeline on faith alone should check this benchmark before spending more compute on it.