Researchers have a fix for one of federated learning's quieter bottlenecks: client devices stepping on each other's gradient math.
Federated training lets many devices or organizations train a shared model without pooling raw data, but it strains memory and bandwidth. LoRA-style adapters cut those costs with a fixed low-rank budget, which caps how much the model can adapt. A newer approach lets each client compress its own gradients into a low-rank subspace instead, but the paper finds that letting clients pick those subspaces independently causes what it calls "subspace fragmentation" - mismatched local projections bias the combined update, especially when client data is uneven. FedLore's fix is to have every client use the same shared low-rank basis within a round, then rotate that basis across rounds so the model's total update isn't stuck under the per-round rank cap. On vision and language tests, including training a model from scratch in federated fashion, the authors report FedLore beating the low-rank adapter baselines it was tested against and matching or exceeding full-parameter training, while still using less bandwidth and optimizer memory.
Federated learning's selling point is training on data that can't leave the device - hospitals, phones, that kind of thing - but only if coordinating clients doesn't cost more than it saves. Most efficiency fixes trade away some accuracy. This one claims to avoid that trade by removing a specific source of bias in how updates get combined, rather than just shrinking the updates further.
It's a single preprint with the authors' own benchmarks, not an independent replication, so the real test is whether FedLore holds up outside hand-picked vision and language tasks.