A new paper argues the tests used to grade time series forecasting models are checking the wrong kind of failure.
Researchers publishing on arXiv say today's benchmarks mostly reward low error on held-out data, and even robustness studies lean on simple distortions: Gaussian noise, random masking, or bounded adversarial tweaks. Real-world breakdowns look different. A failed sensor, a sudden regime shift, or a broken link between two correlated variables can quietly wreck a forecast without ever resembling random noise. The authors propose scenario-grounded stress testing instead: pair each test case with a specific semantic scenario, an explicit failure operator, and a measurable difficulty level, rather than just a single accuracy number.
That distinction matters more as forecasting foundation models spread into transportation, energy, finance, and healthcare systems. These models often train on massive, unaudited datasets, which makes the usual trick of checking accuracy on held-out data less trustworthy as a stand-in for real resilience. Knowing a model is accurate on average is not the same as knowing it survives a faulty sensor or a sudden regime shift.
It is a framework and an argument, not a finished benchmark. The real test will be whether anyone builds and adopts the scenario library the authors are calling for.