Mitigating Update Conflict in Non-IID Federated Learning via Orthogonal Class Gradients

Abstract

The increasingly popular federated learning still faces the practical challenge of non-independent and identically distributed data. Most efforts to address this issue focus on limiting local updates or enhancing model aggregation. However, these methods either restrict the learning capacity of local models or overlook the negative knowledge transfer caused by local objective divergences. In contrast, we observe that the global update can be re-expressed as a weighted sum of the gradients of samples from different classes. Therefore, we hypothesize that the competition among local updates may arise from the conflict between the gradients of samples belonging to different classes. Inspired by this insight, we introduce the novel perspective of orthogonal class gradients, aimed at eliminating interference between updates from different classes without the aforementioned drawbacks. To this end, this paper presents FedOCF, which implements orthogonal class gradient constraints by encouraging orthogonality among features of different classes. Specifically, FedOCF maintains a generator to learn features that are orthogonal for different classes and utilizes it to regularize features learned in local learning. Theoretically, we also demonstrate that FedOCF can improve generalization performance through feature conditional distribution alignment during local learning. Extensive experiments validate the excellent performance of FedOCF in various heterogeneous scenarios.