AdLU: Adaptive double parametric activation functions

Abstract

Activation functions are critical components of neural networks, introducing the necessary nonlinearity for learning complex data relationships. While widely used functions such as ReLU and its variants have demonstrated notable success, they still suffer from limitations such as vanishing gradients, dead neurons, and limited adaptability at various degrees. This paper proposes two novel differentiable double-parameter activation functions (AdLU${}_1$ and AdLU${}_2$) designed to address these challenges. They incorporate tunable parameters to optimize gradient flow and enhance adaptability. Evaluations on benchmark datasets, MNIST, FMNIST, USPS, and CIFAR-10, using ResNet-18 and ResNet-50 architectures, demonstrate that the proposed functions consistently achieve high classification accuracy. Notably, AdLU${}_1$ improves accuracy by up to 5.5 % compared to ReLU, particularly in deeper architectures and more complex datasets. While introducing some computational overhead, their performance gains establish them as competitive alternatives to both traditional and modern activation functions.