Distribution-modulated binary neural network for image classification

Abstract

Deep neural networks excel at image processing tasks, but their extensive model storage and computational overhead make deployment on edge devices challenging. Binary neural networks (BNNs) have become one of the most prevailing model compression approaches due to the advantage of memory and computation efficiency. However, there exists a large performance gap between BNNs and their full-precision counterparts due to training difficulties. When training BNNs using pseudo-gradients, both dead weights and susceptible weights hinder the optimization of BNNs. To solve these two abnormal weights, in this paper, we propose a distribution-modulated binary neural network (DM-BNN), which incorporates a new regularization for dead weights (RDW) and a novel approximation with a peak-shaped derivative (APSD) for susceptible weights. In detail, RDW can supply additional gradients to eliminate dead weights and form a compact weight distribution, while APSD reduces the number of susceptible weights by facilitating the magnitude increase of susceptible weights. The achieved state-of-the-art experimental results on CIFAR-10 and ImageNet demonstrate the effectiveness of DM-BNN. Our code will be available at https://github.com/NianKong/DM-BNN.