Effect of continuous S-shaped rectified linear function on deep convolutional neural network

Abstract

The vanishing gradient issue in convolutional neural networks (CNNs) is often addressed by improving activation functions, such as the S-shaped rectified linear activation unit (SReLU). However, SReLU can pose challenges in updating training parameters effectively. To mitigate this, we propose applying the Aggregation Fischer–Burmeister (AFB) function to SReLU, which smooths the secant line slope of the function from both sides. However, direct application of AFB to SReLU can intensify the vanishing gradient issue due to irregular function behavior. To address this concern, we introduce a regulated version of AFB (ReAFB) that ensures proper gradient and mean activation output conditions when applied to SReLU (ReAFBSReLU). We evaluate the performance of CNNs using ReAFBSReLU on three benchmark datasets: MNIST, CIFAR-10 (with and without data augmentation), and CIFAR-100. Specifically, we implement Network in Network (NIN) for MNIST and CIFAR-10, and LeNet for CIFAR-100 dataset. Additionally, we utilize SqueezeNet exclusively to compare the performance of CNNs using the proposed ReAFBSReLU activation function against state-of-the-art activation functions. Our results demonstrate that ReAFBSReLU outperforms other activation functions tested in this study, indicating its efficacy in enhancing training parameter updates and subsequently improving accuracy.