Sparse loss-aware ternarization for neural networks

Abstract

Deep neural networks (DNNs) have shown great success in machine learning tasks and widely used in many fields. However, the substantial computational and storage requirements inherent to DNNs are usually high, which poses challenges for deploying deep learning models on resource-limited devices and hindering further applications. To address this issue, the lightweight nature of neural networks has garnered significant attention, and quantization has become one of the most popular approaches to compress DNNs. In this paper, we introduce a sparse loss-aware ternarization (SLT) model for training ternary neural networks, which encodes the floating-point parameters into $\{-1,0,1\}$. Specifically, we abstract the ternarization process as an optimization problem with discrete constraints, and then modify it by applying sparse regularization to identify insignificant weights. To deal with the challenges brought by the discreteness of the model, we decouple discrete constraints from the objective function and design a new algorithm based on the Alternating Direction Method of Multipliers (ADMM). Extensive experiments are conducted on public datasets with popular network architectures. Comparisons with several state-of-the-art baselines demonstrate that SLT always attains comparable accuracy while having better compression performance.