Researchers submitted a cancer-staging model to SMM4H-HeaRD 2026 and found that a classical machine learning method outperformed neural networks on the task.
The CaresAI team framed TNM staging — classifying tumor size (T), lymph node involvement (N), and metastasis (M) — as three separate multi-label classification problems using Cancer Genome Atlas pathology reports. They tested TF-IDF features alongside embeddings from ClinicalBERT, BioBERT, and PubMedBERT, pairing each with Logistic Regression, LightGBM, Feed-Forward Neural Networks, and Wide Residual Networks. LightGBM with TF-IDF topped the training phase: AUROC of 0.9368 (T), 0.9524 (N), and 0.8311 (M), with F1-scores of 0.7559, 0.7384, and 0.7017. Wide Residual Networks trailed at 0.839, 0.8502, and 0.803 AUROC with F1-scores of 0.622, 0.702, and 0.9337.
The test results tell a more complicated story. On test set 1, Macro-F1 scores reached 0.978 (T), 0.957 (N), and 0.879 (M) — then fell to 0.807, 0.767, and 1.0 on test set 2, with overall Macro-F1 dropping from 0.938 to 0.858. The authors attribute the gap to class imbalance, lengthy document handling, and limited generalizability. That gap matters most: a staging model that performs differently across two held-out sets cannot be trusted in a clinical workflow where consistency is everything.
That LightGBM with bag-of-words features outpaces purpose-built clinical language models is a useful reminder that benchmark conditions — curated corpora, shared task splits — flatter models in ways real hospital data rarely will.