Self-attentive mechanism-based supervised comparative learning

Abstract

To address the intra-class diversity and inter-class similarity issues in traditional contrast learning, this paper proposes a supervised contrast learning based on a self-attentive mechanism that can effectively increase the feature extraction ability. The proposed method consists of two stages: feature encoder pre-training and linear classifier fine-tuning. In the feature encoder pre-training phase, the supervised contrast loss exploits the labeling information of the data to minimize the distance between similar images in the embedding space and maximize features of different categories as far away as possible, enhancing the effect of contrast learning. Beyond that, the self-attentive mechanism-based block is introduced in the encoder module to explicitly build the interdependence between the convolutional feature channels and further improve the feature learning capability of the model. In the linear classifier fine-tuning stage, parameters of pre-trained encoder are fixed and only the classifier is fine tuned for the downstream classification task. Experiments on the CIFAR-10 and CIFAR-100 datasets demonstrate the superior of our proposed method.