Garbage labels in, garbage model out — but now there may be an automated way to catch the garbage first.
Researchers have published a method that watches how a deep learning network's loss function behaves over multiple training epochs and uses those patterns to flag images that were probably labeled wrong. They tested it on a fundus image dataset used for diabetic retinopathy screening. When 6% of 10,788 gold-standard labels were deliberately flipped, the method caught 75% of the bad labels while generating false positives on fewer than 5% of the correctly labeled images. Correcting the flagged samples and retraining pushed test accuracy from 95.93% to 96.50% — nearly matching the 96.57% ceiling achievable with perfectly clean labels.
The labeling problem is not a corner case. The paper cites estimates that up to 10% of manually labeled medical images carry incorrect annotations, a meaningful drag on any model trained on them. A tool that can triage which images need a second human look — rather than forcing experts to re-examine entire datasets — changes the economics of dataset curation significantly.
The catch is that 75% recall means one in four bad labels still slips through, so this is a filter, not a fix. Still, for a domain where a misclassified retinal scan can delay a diabetes diagnosis, shaving label noise from 6% down to 1.5% with a mostly automated pass is worth taking seriously.