A systematic and effective parameter and network tuning method by utilizing Jacobian rank deficiency

Most of neural network applications rely on the fundamental approximation property of feed-forward networks. In a realistic problem setting, a mechanism is needed to devise a learning process for implementing this approximate mapping based on available data, starting from choosing an appropriate set of parameters in order to avoid overfitting, to an efficient learning algorithm measured by computation and memory complexities, as well as the accuracy of the training procedure, and not forgetting testing and cross-validation for generalization. In the present paper we develop a comprehensive procedure to address the above issues in a systematic manner. This process is based on a common observation of Jacobian rank deficiency. A new numerical procedure for solving the nonlinear optimization problem in supervised learning is introduced which not only reduces the training time and overall complexity but also achieves good training accuracy and generalization.