Comparison of Various Learning Rate Scheduling Techniques on Convolutional Neural Network

Abstract

The learning rate is a hyperparameter which determines how much the model should change concerning the error each time the model parameters are updated. It is important to tune the learning rate properly because if it is set too low, our model will converge very slowly and if set too high, our model may diverge from the optimal error point. Some conventional learning rate tuning techniques include constant learning rate, step decay, cyclical learning rate and many more. In this paper, we have implemented some of these techniques and compared the model performances gained using these techniques.