Iterative knowledge distillation and pruning for model compression in unsupervised domain adaptation

Abstract

In practical applications, deep learning models often face the challenges of inconsistent distribution between training data and test data and insufficient labeled data. To address these problems, unsupervised domain adaptation (UDA) based transfer learning has gained significant attention. However, the existing UDA models are difficult to meet the requirements of real-time and resource-constrained scenarios. Although model compression can accelerate UDA, it usually leads to performance degradation. In this paper, we propose an iterative transfer model compression (ITMC) method, which centers on two key modules, i.e., transfer knowledge distillation (TKD) and adaptive channel pruning (ACP), by executing them alternately. The tight coupling of the two modules realizes the effective compression of the model while ensuring the performance of the model on the target domain. In the TKD phase, the teacher model and the student model are gradually adapted to the target domain, and the real-time updated teacher model efficiently guides the student model learning, while the ACP phase employs a dynamic pruning strategy based on the training epoch, which removes unimportant channels based on the loss of the TKD student model. Experimental results demonstrate that ITMC approach achieves higher accuracy under the same compression ratio compared with the state-of-the-art methods.