A new technique lets tabular foundation models stop ignoring the subgroups they predict worst, without retraining the whole model.
The method, called DR-TFM, is a parameter-efficient way to make tabular foundation models more reliable when group proportions shift between training and real-world use. Instead of touching the whole network, it fine-tunes or adds a small query-scaling component that adjusts how the model weighs labeled context examples, leaving everything else frozen. On TabPFN-3, that means updating just 0.016% of the model's parameters. Crucially, it does this without needing true labels for which group each example belongs to, estimating groups or source distributions from the training data instead. Across five tabular benchmarks, DR-TFM beat both the original pretrained models and existing robust baselines on worst-group accuracy, while keeping overall mean accuracy competitive. It also improved worst-group accuracy on the ACS Income benchmark and across four other tabular foundation models.
Tabular models increasingly drive decisions on income, credit, and benefits, exactly the kind of predictions where a model can look accurate on average while quietly failing a specific population. Most fixes for this problem assume you know which examples belong to which group, a labeling task that is often impossible or legally fraught in production data. DR-TFM's selling point is doing the correction without that information, and at a fraction of the usual fine-tuning cost.
Updating 0.016% of a model's parameters is cheap, but the gains show up on curated academic benchmarks, not the messier production data where defining an underrepresented group is half the problem.