Robustness Analysis of Neural Network Designs for ReLU Family and Batch Normalization

Abstract

The so-called neural network (NN) robustness problem, its original definition is that if an image is perturbed, the classification result of the image can still maintain the original correct category. It means that at the end of the entire NN operation, the lower bound of the target category value must be greater than the upper bound value of all other categories. There are multiple design techniques which can either bring the neural networks higher accuracy or maintain accuracy while reducing the computation effort at the same time. However, very few work focus on giving an efficient and reliable estimation of the trend of robustness changing directly with respect to the design factors. Lacking such information would either damage on designing a robust NN for critical systems or postpone the robustness analysis after the design completed and then results in paying more cost on NN design modifications. In this paper, we not only provide numerous experimental results but also propose three extended lemmas based on the related work which analyzes robustness with Lipschitz constant, to discuss how the two commonly used design factors, the ReLU based activation functions and batch normalization technique, bring the effectiveness to the robustness changing trend, under the condition of that the NN can still retain acceptable accuracy. We can conclude that we encourage to adopt ReLU than its other family activation functions (e.g. Leaky-ReLU and ELU) but discourage to use batch normalization compared with adopting it in the same NN design if we expect for higher robustness.