Researchers have found that federated learning systems can't rely on a single "is this data bad" signal - the right one depends on what kind of corruption you're looking for.
A new arXiv paper tests two ways of flagging bad data in federated learning: uncertainty estimates (how confused a model is about an input) and loss (how badly a prediction matches its assigned label). Testing on ResNet-20 with the CIFAR-10 and SVHN datasets, split unevenly across simulated clients, the researchers introduced two corruption types: persistent label flips and added image noise. For label flips, loss-based detection scored well (AUC of 0.85 on CIFAR-10, 0.95 on SVHN), while every uncertainty measure tested performed no better than a coin flip. For image noise, the pattern flipped: an uncertainty measure called expected entropy outperformed loss, with its edge growing as corruption spread further across the federation.
Federated learning promises privacy by training models on data that never leaves users' devices, but that also makes bad data hard to spot centrally. This research shows why one popular fix, using model uncertainty as a catch-all corruption detector, quietly fails on mislabeled data even though it works fine on noisy images. Teams building federated systems for phones, hospitals, or other distributed data sources need to run both checks rather than assume one covers everything.
It is a reminder that "detect bad data" is not one problem. A model can be dead certain about a photo and still have been handed the wrong label for it.