A new benchmark exposes a flaw in how researchers measure federated learning pipelines trained on 3D point cloud data.
Researchers ran 504 training experiments combining 13 federated learning algorithms with 10 knowledge distillation objectives, testing across a standard 3D object dataset and a clinical skull-imaging dataset. The results confirmed what practitioners suspected: federated learning struggles badly when data is unevenly distributed across devices. The best federated model hit 76.32% accuracy on the object dataset against a 92.26% ceiling from centralized training - and only 75.83% on the clinical data, where centralized training achieves a perfect 100%. Knowledge distillation helped on the compression side, shrinking models by 74.51% while roughly doubling inference speed, often without losing accuracy relative to the original teacher model.
The more consequential finding is an evaluation trap buried in standard practice. When a distillation objective includes a hard-label cross-entropy term - meaning the student learns partly from labeled proxy data rather than purely from the federated teacher - a collapsed federated teacher scoring 8.50% can still produce a student at 92.94%. That 84-point gap does not reflect what the federated model learned; it reflects the proxy labels. Those labels, notably, are the exact data whose privacy concerns motivated using federated learning in the first place, making the setup circular.
This is a methodological warning for anyone building or evaluating privacy-preserving ML systems at the edge. The benchmark recommends label-free distillation objectives so reported accuracy actually tracks the federated model's quality rather than laundering centralized labels through a student network. Medical imaging and autonomous perception - two fields where federated learning is most loudly promoted - are also where inflated benchmarks cause the most damage.