Reducing weight divergence impact using local learning normalization in Federated Learning for heterogeneous data distributions

In an increasingly connected world, technologies such as smartphones, 5G, drones, the Internet of Things, and Smart Cities bring new challenges and opportunities. The increase in data collected by these devices and their ease of access allows the use of machine learning techniques to provide intelligent and quality services. Considering these services’ distributed access to data, using Federated Learning is an excellent option for decentralized machine learning processing with greater security. However, client access to heterogeneously distributed data impacts federated learning, reducing test accuracy and increasing communication costs. To address these problems, we present a new method called Federated Learning with Weight Standardization on Convolutional Neural Networks (FedWS) that uses normalization on weights in local training on convolutional neural networks, optimizing the training gradients and reducing the impact of weight divergence caused by heterogeneous distributions on federated training tasks. Results showed that our method got superior results on image classification tasks in the order of 1.36% to 5.0% of test accuracy at different levels of heterogeneity. Their behavior showed a reduction in the effects of divergence at higher levels of heterogeneity and additional communication cost reduction ranging from 20% to 100% due to fast convergence and being more suitable for use in mobile devices with computational resource limitations.