Using Knowledge Transfer for Neural Network Architecture Optimization with Improved Training Efficiency

Abstract

Neural networks have demonstrated their potential for solving a wide range of real-world problems. While capable of learning correlations and relationships among complex datasets, neural network training and validation are often tedious and time-consuming, especially for large, high-dimensional datasets, given that sufficient models should be run through the validation pipeline so as to adequately cover the hyperparameter space. In order to address this problem, we propose a weight adjustment method for improving training times when searching for the optimal neural network architecture by incorporating a knowledge transfer step into the corresponding pipeline. This involves learning weight features from a trained model and using them to improve the training times of a subsequent one. Considering a sequence of candidate neural networks with different hyperparameter values, each model, once trained, serves as a source of knowledge for the next one, resulting in improved overall training times. We test our approach on dataset of various sizes, where we obtain reduced total training times, especially when small learning rates are used for finely-tuned convergence.