A unified DNN weight compression framework using reweighted optimization methods

Abstract

To address the large model sizes and intensive computation requirements of deep neural networks (DNNs), weight pruning techniques have been proposed and generally fall into two categories: static regularization-based pruning and dynamic regularization-based pruning. However, the static method often leads to either complex operations or reduced accuracy, while the dynamic method requires extensive time to adjust parameters to maintain accuracy while achieving effective pruning. In this paper, we propose a unified robustness-aware framework for DNN weight pruning that dynamically updates regularization terms bounded by the designated constraint. This framework can generate both non-structured sparsity and different kinds of structured sparsity, and it incorporates adversarial training to enhance the robustness of the sparse model. We further extend our approach into an integrated framework capable of handling multiple DNN compression tasks. Experimental results show that our proposed method increases the compression rate – up to $630\times$ for LeNet-5, $45\times$ for AlexNet, $7.2\times$ for MobileNet, $3.2\times$ for ResNet-50 – while also reducing training time and simplifying hyperparameter tuning to a single penalty parameter. Additionally, our method improves model robustness by 5.07% for ResNet-18 and 3.34% for VGG-16 under a 16×pruning rate, outperforming the state-of-the-art ADMM-based hard constraint method.