FWC: Fitting Weight Compression Method for Reducing Communication Traffic for Federated Learning

Abstract

Federated learning enables local nodes to train a global model together by uploading only training updates to the parameter server without exchanging private data. However, as the complexity of the federated learning task increases, the communication volume of the training process becomes extremely large, hence the huge communication traffic becomes a serious bottleneck in current federated learning application. Existing methods reduce communication overhead from two aspects, the number of communications and the traffic per communication. But these methods usually lead to more consumption of computing resources or a decrease in model accuracy. To handle these problems, this paper proposes a data fitting based weight compression algorithm, FWC, which includes four sequential stages: sparsification, polynomial fitting, encoding, reconstruction and two mechanism: warm-up and accumulation. In particular, the warm-up mechanism can well address the problem of slow convergence in early training period. Experimental results on models with different scales show that FWC is able to provide more than 600x traffic compression at the cost of only millisecond-level computational time cost and less than 1% accuracy loss.