FNCSE: contrastive learning for unsupervised sentence embedding with false negative samples

Abstract

In recent years, contrastive learning has gradually been applied to the field of natural language processing from the explosion of the computer vision community. Generally speaking, for a given sentence, the original sentence is used as the anchor point, the current model uses a special data enhancement method to generate positive samples, and the remaining sentences in the batch are used as negative samples for the sentence. Although this approach effectively improves sentence embedding, we believe that in the process of constructing negative samples, negative samples will also have information similar to the current anchor point, and should not be mistakenly attributed to negative samples, which will confuse the learning ability of the model, and The increase in the number of negative pairs during training is beneficial to the training of the model. In this paper, we propose methods to identify false negatives in the text domain, and two strategies to mitigate the impact of false negatives: masking false negatives and adding to the set of positives. In addition, we propose an optimized version of momentum contrast to expand the number of negative pairs. Our method is based on the improvement made on SimCSE called FNCSE. We evaluate FNCSE on multiple benchmark datasets in the semantic similarity (STS) task. The experimental results show that in the Bert base, FNCSE averages 1.29% Spearman Correlation is better than SimCSE.