A new study says a decade of health AI research hasn't produced consistent winners.
Researchers re-implemented 12 historical and recent algorithms for predicting outcomes from electronic health records, then tested them in a shared framework on two clinical datasets, MIMIC-IV and NWICU. They compared expert-authored clinically meaningful tasks against tasks generated from randomly sampled event codes and prediction horizons. Algorithm rankings held up consistently across both task types and both datasets. But on the clinically meaningful tasks, the best method depended more on the specific task, suggesting some modeling choices only pay off in particular scenarios.
The bigger finding undercuts the assumption that newer, more complex approaches automatically beat older ones: gradient-boosted trees, paired with a modern, wide and sparse representation of EHR data, stayed highly competitive with the newest algorithms. That's a useful check for a field where flashy benchmarks often justify swapping in the latest architecture. For hospitals and health-tech vendors deciding what to build on, boring and well-understood models may still be the safer bet.
It's the health AI equivalent of discovering your spreadsheet macros still outperform the fancy new dashboard: unglamorous, but it works.