A new study says most of the transformer blocks inside tabular foundation models are dead weight.
Researchers tested eight frontier tabular foundation models (TFMs) across 171 classification datasets using a label-free technique that swaps out up to 96% of a model's transformer blocks with a simple closed-form affine translator. Cutting half the blocks changed median AUROC by at most 0.002, and the shrunk models still beat full-depth TFMs and tuned gradient-boosted decision trees (GBDTs). Different architectures hid their redundancy differently: TabPFN-3 spread its capability across encoders and kept working no matter which single block survived, while other models leaned on just their first block or packed their work into one critical block. The cuts freed up to 1.5 billion parameters and sped GPU inference by up to 6x, with a single-block TabPFN-3 serving cached queries 2.4x faster using 6x less memory.
Tabular foundation models were pitched as the lightweight option for privacy-sensitive or resource-constrained settings precisely because they skip extra training. If most of their depth turns out to be unnecessary, the computational case for reaching for a transformer over a well-tuned GBDT gets a lot weaker, especially since this fix needs no retraining, just removing blocks.
GBDTs have been "good enough" for tabular data for over a decade; this study mostly explains why the fancier alternative kept overshooting.