Tabular foundation models look good on paper — until you hand them a harder test.
Researchers introduced BeyondArena, a unified benchmark designed to evaluate foundation models on tabular data across conditions that standard benchmarks routinely skip. The study tested 11 models against 142 curated datasets, spanning not just standard independent and identically distributed data but also temporal splits, grouped data, large sample sizes, and high-dimensional feature sets. The team also released Data Foundry, a Python framework for curating tabular datasets with a shared metadata schema, meant to reduce the fragmentation that has kept discipline-specific evaluations siloed from model researchers.
The findings cut against the hype: foundation models for tabular data perform well on small- to medium-sized IID datasets — exactly the conditions where they have always been tested. On non-IID, large, or high-dimensional data, traditional tree-based models and standard deep learning approaches still win. That gap matters because the easy cases are largely solved; the hard cases are where real-world tabular data tends to live.
The benchmark community has seen this pattern before with language and vision models, where progress on curated leaderboards stalled until harder out-of-distribution tests forced a reckoning. Tabular AI is apparently running the same script, just a few years behind.