Learning rate range test for the vision transformer

Abstract

The solutions obtained by training the deep neural network are highly dependent on the parameters including the learning rate. Therefore, finding the appropriate settings for training deep neural networks is very important. In particular, it is necessary to find the better settings for SOTA models of Vision Transformer(ViT), whose structure is different from ordinal models. In this paper, we focus on the learning rate to find a better value using the Learning Rate Range Test (LRRT). Through our experiments, we found that the appropriate LR is located where the decrease in loss value stops in the LRRT. In addition, we discuss about the effects of the number of epochs and the LR warm up.