A new guide lays out, in granular detail, how machine learning researchers keep fooling themselves with bad validation splits.
The tutorial reviews the standard menu of validation techniques, hold-out sets, k-fold and repeated stratified cross-validation, leave-one-out schemes, and group-aware and nested group cross-validation, then maps them onto real biomedical data types like EEG epochs, paired-eye OCT scans, repeated clinical measurements, and multicenter datasets. It builds eight controlled scenarios comparing flawed designs against leakage-safe ones, covering problems like global feature selection done before splitting, normalization leakage, dependent records split across train and test, and center mixing in multi-site data. Seven scenarios use locked confusion matrices with auditable metrics; one uses a repeated-study simulation. The authors also release MATLAB and scikit-learn templates, a data-size matrix, a decision tree, and reporting checklists.
This matters because data leakage is the quiet killer of reproducibility in applied ML, especially in medicine, where an inflated validation score can make a model look clinically useful when it is not. The paper's central point is unglamorous but important: there is no universally correct validation method, only one that matches how the model will actually be deployed, down to the level of which records are allowed to share information with each other.
None of this is new in principle, researchers have warned about leakage for years, but a scenario-by-scenario accounting with locked, auditable results is the kind of unsexy reference work that tends to get cited quietly for a decade rather than celebrated for a week.