The New Activation Function for Complex Valued Neural Networks: Complex Swish Function

Abstract

Complex-valued artificial neural network (CVANN) has been developed to process data with complex numbers directly. Weights, threshold, inputs and outputs are all complex numbers in the CVANN. The convergence of the CVANN back propagation algorithm depends on some factors such as selection of appropriate activation function, threshold values, initial weights and normalization of data. The most important of these factors is the selection of the appropriate activation function. The selection of activation function determines the convergence and general formation characteristics of the complex back propagation algorithm. In this study, the swish activation function discovered by Google researchers Prajit Ramachandra, Barret Zoph and Quoc V. Le is discussed in the complex domain. Swish activation function, which gives good results in real plane, has been studied in the complex plane. We have compared the performance of swish activation functions on the complex XOR and symmetry problems with other known activation functions. The simulations’ results show that the proposed network using swish activation function, gives the best results when compared to other networks using the traditional complex logarithmic sigmoid and tangent sigmoid activation functions.