DCLCSE: Dynamic Curriculum Learning Based Contrastive Learning of Sentence Embeddings

Abstract

Recently, Contrastive Learning (CL) has made impressive progress in natural language processing, especially in sentence representation learning. Plenty of data augmentation methods have been proposed for the generation of positive samples. However, due to the highly abstract nature of natural language, these augmentations cannot maintain the quality of generated positive samples, e.g., too easy or hard samples. To this end, we propose to improve the quality of positive examples from a data arrangement perspective and develop a novel model-agnostic approach: Dynamic Curriculum Learning based Contrastive Sentence Embedding framework (DCLCSE) for sentence embeddings. Specifically, we propose to incorporate a curriculum learning strategy to control the positive example usage. At the early learning stage, easy samples are selected to optimize the CL-based model. As the model's capability increases, we gradually select harder samples for model training, ensuring the learning efficiency of the model. Furthermore, we design a novel difficulty measurement module to calculate the difficulty of generated positives, in which the model's capability is considered for the accurate sample difficulty measurement. Based on this, we develop multiple arrangement strategies to facilitate the model learning process based on learned difficulties. Finally, extensive experiments over multiple representative models demonstrate the superiority of DCLCSE. As a byproduct, we have released the codes to facilitate other researchers.