An Empirical Study on Unsupervised Pre-training Approaches in Regression Problems

Abstract

Unsupervised pre-training allows for efficient training of deep architectures. It provides a good set of initialised weights to the deep architecture that can provide better generalisation of the data. In this paper, we aim to empirically analyse the effect of different unsupervised pre-training approaches for the task of regression on different datasets. We have considered two most common pre-training methods namely deep belief network and stacked autoencoder, and compared the results with the standard training algorithm without pretraining. The models with pretraining performed better than the model without pretraining in terms of error, convergence and the prediction of pattern. The results of the experiments also show the importance of hyperparameters tuning, specially learning rate, in providing a better prediction result. This study once again confirmed the effectiveness and potential of pretraining approach in nonlinear regression problem.