Reducing communication overhead through one-shot model pruning in federated learning

Abstract

In the realm of federated learning, a collaborative yet decentralized approach to machine learning, communication efficiency is a critical concern, particularly under constraints of limited bandwidth and resources. This paper evaluates FedSNIP, a novel method that leverages the SNIP (Single-shot Network Pruning based on Connection Sensitivity) technique within this context. By utilizing SNIP, FedSNIP effectively prunes neural networks, converting numerous weights to zero and resulting in sparser weight representations. This substantial reduction in weight density significantly decreases the volume of parameters that need to be communicated to the server, thereby reducing the communication overhead. Our experiments on the CIFAR-10 and UCI-HAR dataset demonstrate that FedSNIP not only lowers the data transmission between clients and the server but also maintains competitive model accuracy, comparable to conventional federated learning models. Additionally, we analyze various compression algorithms applied after pruning, specifically evaluating the compressed sparse row, coordinate list, and compressed sparse column formats to identify the most efficient approach. Our results show that compressed sparse row not only compresses the data more effectively and quickly but also achieves the highest reduction in data size, making it the most suitable format for enhancing the efficiency of federated learning, particularly in scenarios with restricted communication capabilities.