A new benchmarking study finds that judging class-imbalance fixes on a single dataset can produce the exact opposite of the right answer.
The researchers ran a leakage-free nested cross-validation protocol, selecting the decision threshold on a held-out validation fold, on the Kaggle credit-card fraud dataset, a favorite benchmark for imbalance research. At the default 0.5 threshold, a Random Forest scored F1 = 0.861, and tuning the threshold added nothing (delta-F1 = -0.002) - seemingly proof that a well-calibrated model doesn't need imbalance handling. But when the team repeated the identical protocol across 45 binary classification tasks with imbalance ratios from 1:1.5 to 1:178 (2,025 model fits, four model families), Random Forest's result reversed: it benefited most from threshold tuning of any family tested (delta-F1 = +0.101), while the other three families behaved on the wider suite much as they had on the fraud set alone. SMOTE oversampling, which hurt performance on the fraud dataset, helped across the suite instead, winning far more often than it lost (138 wins vs. 39 losses).
The fraud dataset's imbalance ratio, 1:577, sits well outside the range where threshold tuning actually pays off - the benefit peaks between 1:15 and 1:40 and fades past 1:100, so testing on that one dataset alone was close to guaranteed to mislead. The study also knocks out a popular shortcut: calibration error, the metric practitioners lean on to decide whether tuning is worth trying, does not predict whether it will help.
Imbalance-handling papers built on a single Kaggle benchmark are common enough to be a genre. This one is a reminder that a benchmark isn't a law of nature, and a result that only holds on one dataset might just be describing that dataset.