Data-specific activation function learning for constructive neural networks

Abstract

Activation functions play a crucial role in learning and expressive capabilities of advanced neural networks due to their non-linear or non-saturated properties. However, how to determine the appropriate activation function from various candidates is a challenging yet not well-addressed topic. To address the issue, a novel self-learning approach, called as data-specific activation function learning (DSAFL) algorithm, is proposed to establish constructive neural network on one-time by adaptively selecting appropriate activation function based on the specific data characteristics. To assess the space dimension mapping abilities of different activation functions, the configuration probabilities are used to guide the generation of various candidate activation functions and corresponding candidate hidden node. In the learning stage, an exploration-exploitation mechanism composed of the random algorithm and the greedy strategy is developed to obtain the influence of different candidate activation functions, thereby avoiding configuration probabilities falling into local optimum. A reward-penalty mechanism is built to update the configuration probabilities and enhance the robustness of network by integrating the simulated annealing strategy. In final, the activation function with the highest configuration probability, as the best one, is used to reconstruct the neural network. Experimental results on both regression and classification tasks demonstrate the efficiency and effectiveness of DSAFL in the activation function selection problems of a class of constructive neural networks.