The Role of Activation Function in CNN

Abstract

We all know that the purpose of introducing activation function is to give neural network nonlinear expression ability, so that it can better fit the results, so as to improve the accuracy. However, different activation functions have different performance in different neural networks. In this paper, several activation functions commonly used by researchers are compared one by one, and qualitative comparison results are given by combining with specific neural network models. For example, when using the MNIST dataset in LeNet, PReLU achieved the highest accuracy of 98.724%, followed by Swish at 98.708%. When cifar-10 data set was used, the highest accuracy rate of ELU was 64.580%, followed by Mish at 64.455%. When Using VGG16, ReLU reached the highest accuracy of 90.226%, followed by PReLU at 90.197%. When using ResNet50, ELU achieved the highest accuracy of 89.943%, followed by Mish at 89.780%.