Dynamic feedforward network architecture design based on information entropy

Abstract

To solve the problem of the neural network architecture design, a dynamic feedforward neural network architecture design method based on information entropy is proposed. In this method, the neural network's cost function is composed of the cross entropy of the neural network's expected output and actual output and Renyi's entropy of the hidden node's output. This does not require the learning samples to obey the Gauss distribution. In the learning processing, the bumber of the hidden neurons is dynamically adjusted by splitting the most active hidden neurons and removing the least active hidden neurons. This approach can improve the neural network's dynamic response ability and can solve the problem of self-organizing architecture design of the feedforward neural network. The proposed method is applied to online modeling of ammonia nitrogen in tahe wastewater treatment process based on actual operating data. The experiment illustrates the dynamic response capability and the online learning capacity of the neural network.