Certified Accuracy and Robustness: How different architectures stand up to adversarial attacks

Abstract

Adversarial attacks are a concern for image classification using neural networks. Numerous methods have been created to minimize the effects of attacks, where the best defense against such attacks is through adversarial training, which has proven to be the most successful to date. Due to the nature of adversarial attacks, it is difficult to assess the capabilities of a network to defend. The standard method of assessing a network’s performance in supervised image classification tasks is based on accuracy. However, this assessment method, while still important, is insufficient when adversarial attacks are included. A new metric called certified accuracy is used to assess network performance when samples are perturbed by adversarial noise. This paper supplements certified accuracy with an abstention rate to give more insight into the network’s robustness. Abstention rate measures the percentage of the network that failed to keep its prediction unchanged as the perturbation strength increases from zero to specified strength. The study focuses on popular and good-performing CNN-based architectures, specifically EfficientNet-B7, ResNet-50, ResNet-101, Wide-ResNet-101, and transformer architectures such as CaiT and ViT-B/16. The selected architectures are trained in adversarial and standard methods and then certified on CIFAR-10 datasets perturbed with Gaussian noises of different strengths. Our results show that transformers are more resilient to adversarial attacks than CNN-based architectures by a significant margin. Transformers exhibit better certified accuracy and tolerance against stronger noises than CNN-based architectures, demonstrating good robustness with and without adversarial training. The width and depth of a network have little effect on achieving robustness against adversarial attacks, but rather, the techniques that are deployed in the network are more impactful, where attention mechanisms have been shown to improve a network’s robustness.