A new benchmark grades time series forecasting models on data that hasn't happened yet.
Researchers unveiled LiveHouse-TS, described as the first "living" benchmark for time series foundation models. Unlike standard benchmarks that test models on a fixed historical window, LiveHouse-TS runs prequential evaluation, scoring models continuously against real future data as it arrives. The infrastructure spans 11 domains and 17 datasets, tracking how forecasting accuracy holds up through seasonal swings, distribution shifts, and one-off surprises the training data never saw. The team's streaming tests found that rankings built on static benchmarks reshuffle dramatically once models face live conditions.
Most foundation-model claims, in time series forecasting and elsewhere, lean on leaderboard snapshots that reward memorizing a fixed test set rather than adapting to a moving world. If a model that tops today's static benchmark can slide down the rankings within weeks of live data, buyers evaluating forecasting tools for supply chains, energy grids, or finance are getting a number that expires fast. LiveHouse-TS turns benchmarking from a one-time certificate into an ongoing audit, which is a much closer match for how these models actually get used.
It is the forecasting equivalent of grading a weather model on tomorrow's weather instead of last year's, and it is a bar that a lot of current leaderboard leaders may not clear.