Robustness of Compressed Deep Neural Networks with Adversarial Training

Abstract

Deep learning models are not applicable on edge computing devices. Consequently, compressed deep learning models gain momentum recently. Meanwhile, adversarial attacks targeting conventional deep neural networks (DNNs) and compressed DNNs are flouring nowadays. This paper firstly surveys the current compressing techniques, including pruning, distillation, quantization and weights sharing. Then, two iterative adversarial attacks, including I-FGSM (Iterative-Fast Gradient Sign Method) and PGD (Project Gradient Descent), are introduced. Three scenarios are built to test each DNN’s robustness against adversarial attacks. Besides, each DNN is trained with samples generated by different adversarial attacks and is then compressed under different pruning rate and tested under different attacks. The experimental results prove firstly that when a DNN is compressed with pruning rate lower than 70.0% is safe and with tiny accuracy decline. Second, iterative adversarial attacks are effective and cause dramatic performance degradation. Third, adversarial training helps to secure the compressed DNNs while lowering transferability of adversarial samples constructed by different attack algorithms.