A hyper-heuristic enhanced neuro-evolutionary algorithm with self-adaptive operators and various activation functions for classification problems

Abstract

Due to their remarkable generalization capabilities, Artificial Neural Networks (ANNs) grab attention of researchers and practitioners. ANNs have two main stages, namely training and testing. The training stage aims to find optimum synapse values. Traditional gradient-descent-based approaches offer notable advantages in training ANNs, nevertheless, they are exposed to some limitations such as convergence and local minima issues. Therefore, stochastic search algorithms are commonly employed. In this regard, the present study adopts and further extends the evolutionary search strategies namely, the Differential Evolution (DE) algorithm and the Self-Adaptive Differential Evolution (SaDE) Algorithm in training ANNs. Accordingly, self-adaptation procedures are modified to perform a more convenient search and to avoid local optima first. Secondarily, mutation operations in these algorithms are reinforced by a hyper-heuristic framework, which selects low-level heuristics out of a heuristics pool, based on their achievements throughout search. Thus, to achieve better performance, while promising mechanisms are more frequently invoked by the proposed approach, naïve operators are less frequently invoked. This implicitly avoids greedy behavior in selecting low-level heuristics and attempts to overcome loss-of-diversity and local optima issues. Moreover, due to possible complexity and nonlinearity in mapping between inputs and outputs, the proposed method is also tested by using various activation functions. Thus, a hyper-heuristic enhanced neuro-evolutionary algorithm with self-adaptive operators is introduced. Finally, the performances of all used methods with various activation functions are tested on the well-known classification problems. Statistically verified computational results point out significant differences among the algorithms and used activation functions.