ESH: A Non-Monotonic Activation Function For Image Classification

Abstract

By providing non-linearity and enabling the network to understand complicated associations in the data, activation functions play a vital role in the performance of neural networks. Here, we introduce Esh, a brand-new activation function with the formula, $f(x) = xtanh(sigmoid(x))$. Using CNN architectures, we assess Esh’s performance on the MNIST, CIFAR10, and CIFAR-100 data sets. Our tests demonstrate that the Esh activation function outperforms a number of well-known activation functions, including ReLU, GELU, Mish, and Swish. In fact, compared to other activation functions, the Esh activation function has a more consistent loss landscape. Esh is a potential new activation function for deep neural networks, according to the findings of our study, and we anticipate that it will be widely used in the machine learning industry.