Analysis of The Change of Softmax Value in The Training Process of Neural Network

Abstract

Classification is an essential field in deep learning. Generally, the category corresponding to the maximum value of softmax is mainly used as the prediction result and the softmax value as the prediction probability. However, whether softmax can indeed serve as a prediction probability needs further confirmation. This paper first focuses on the classification of paintings through Convolutional Neural Network. To deal with the imbalanced dataset problem, only those with more than 200 paintings are selected. Besides, class weight is also taken into consideration. Next, data augmentation is applied to enlarge the dataset and add more relevant data. For the modeling and training part, transfer learning is employed to avoid training from scratch on a new dataset and reduce the cost of later training. Techniques such as ‘EarlyStopping’ and ‘ReduceLROnPlateau’ are also used to avoid overfitting. The final prediction accuracy can achieve 99 percent on the training and 87 percent on the validation sets. Furthermore, the paper studies the change of softmax distribution during the training process and the relationship between the average maximum value of softmax and the classification performance of classes. The experiments show that the maximum value of softmax will gradually shift to the corresponding correct label during the training process. Still, there is no correlation between the classification performance and the average maximum value of softmax. Therefore, softmax cannot be used as a probability value for classification.