Learning to Binarize Convolutional Neural Networks with Adaptive Neural Encoder

The high computational complexity and memory consumption of the deep Convolutional Neural Networks (CNNs) restrict their deployability in resource-limited embedded devices. To address this challenge, emerging solutions are proposed for neural network quantization and compression. Among them, Binary Neural Networks (BNNs) show their potential in reducing computational and memory complexity; however, they suffer from considerable performance degradation. One of the major causes is their non-differentiable discrete quantization implemented using a fixed sign function, which leads to output distribution distortion. In this paper, instead of using the fixed and naive sign function, we propose a novel adaptive Neural Encoder (NE), which learns to quantize the full-precision weights as binary values. Inspired by the research of neural network distillation, a distribution loss is introduced as a regularizer to minimize the Kullback-Leibler divergence between the outputs of the full-precision model and the encoded binary model. With an end-to-end backpropagation training process, the adaptive neural encoder, along with the binary convolutional neural network, could reach convergence iteratively. Comprehensive experiments with different network structures and datasets show that the proposed method can improve the performance of the baselines and also outperform many state-of-the-art approaches. The source code of the proposed method is publicly available at https://github.com/CQUlearningsystemgroup/LearningToBinarize.