Comparative Analysis of Activation Functions Used in the Hidden Layers of Deep Neural Networks

Abstract

The development in the field of neural networks opens up opportunities for the use of many activation functions, each of which has its own specific features. This raises questions about how compatible the different activation functions are and whether their exchange affects the operation of a neural network. The article reviews the design, training and research of a Deep Neural Network. The Network is applied for curve recognition Three popular activation functions are analysed in the hidden layers – sigmoid function (Sigmoid), a hyperbolic tangent (tanh) and a rectified linear unit (ReLU). The results of this study will be useful in the design of Deep Neural Networks and in the selection of activation functions.