Rectified Tangent Activation (RTA): A Novel Activation Function for Enhanced Deep Learning Performance

In deep learning, activation functions (AFs) influence a model’s performance, convergence rate, and generalization capability. Conventional activation functions such as ReLU, Swish, ELU, and Tanh have been widely utilized, each offering distinct advantages but also exhibiting intrinsic drawbacks. ReLU is computationally efficient but susceptible to the “dying ReLU” phenomenon, whereas Tanh has saturation problems in both its positive and negative ranges. This study presents the Rectified Tangent Activation (RTA) function, an innovative activation function developed to overcome these restrictions by integrating advantageous features of ReLU, Swish, ELU, and Tanh. We assess the efficacy of RTA by a comparison study with five prevalent activation functions: ELU, ReLU, Swish, and Tanh, utilizing four distinct datasets—CIFAR-10, CIFAR-100, Fashion MNIST, and Chest X-ray. The findings demonstrate that RTA regularly attains superior performance, ranking first on the CIFAR-100, Fashion MNIST, and Chest X-ray datasets, while achieving a robust second place on CIFAR-10, trailing only ELU. The versatility of RTA across diverse data sets, such as image classification and medical imaging, underscores its potential as a versatile AF for numerous deep-learning applications. Our findings indicate that RTA can alleviate problems such as gradient saturation and convergence delay while improving overall accuracy. Considering these encouraging outcomes, RTA offers a persuasive alternative for deep learning practitioners aiming for strong model performance with reduced computing demands.