Clean, performance-robust, and performance-sensitive historical information based adversarial self-distillation

Abstract

Adversarial training suffers from poor effectiveness due to the challenging optimisation of loss with hard labels. To address this issue, adversarial distillation has emerged as a potential solution, encouraging target models to mimic the output of the teachers. However, reliance on pre-training teachers leads to additional training costs and raises concerns about the reliability of their knowledge. Furthermore, existing methods fail to consider the significant differences in unconfident samples between early and late stages, potentially resulting in robust overfitting. An adversarial defence method named Clean, Performance-robust, and Performance-sensitive Historical Information based Adversarial Self-Distillation (CPr & PsHI-ASD) is presented. Firstly, an adversarial self-distillation replacement method based on clean, performance-robust, and performance-sensitive historical information is developed to eliminate pre-training costs and enhance guidance reliability for the target model. Secondly, adversarial self-distillation algorithms that leverage knowledge distilled from the previous iteration are introduced to facilitate the self-distillation of adversarial knowledge and mitigate the problem of robust overfitting. Experiments are conducted to evaluate the performance of the proposed method on CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets. The results demonstrate that the CPr&PsHI-ASD method is more effective than existing adversarial distillation methods in enhancing adversarial robustness and mitigating robust overfitting issues against various adversarial attacks.