Optimizing Vision Transformer Performance with Customizable Parameters

Abstract

This paper experimentally examined the effects of changing the size of image patches and the number of transformer layers on the training time and accuracy of a vision transformer used for image classification. The transformer architecture was first introduced in 2017 as a new way of processing natural language and has since found applications in computer vision as well. In this experiment, we trained and tested fourteen versions of a vision transformer on the CIFAR-100 dataset using graphical processing units provided by Google Colaboratory. The results showed that increasing the number of transformer layers and decreasing the patch size both increased test accuracy and training time. However, learning curves generated by the models showed overfitting for very small patch sizes. Overall, changing patch size had a greater impact on accuracy than changing the number of transformer layers. The results also suggested that transformers are more resource-intensive to train than other models. We suppose that including a classification token could lead to shorter training times, but another experiment is needed to examine its influence on accuracy.