Contrastive Learning with Transformer Initialization and Clustering Prior for Text Representation

Abstract

Acquiring labeled data for learning sentence embeddings in Natural Language Processing poses challenges due to limited availability and high costs. In order to tackle this issue, we introduce a novel method called Contrastive Learning with Transformer Initialization and Clustering Prior for Text Representation (CLTC). Our method utilizes Pre-Layernorm Transformers without warm-up, stabilizing the training process while also increasing the final performance. We employ Contrastive Learning (CL) with dropout-based augmentation to enhance sentence embeddings. Additionally, we integrate prior knowledge into the contrastive learning framework within an efficient clustering strategy. When evaluated on the SentEval task, our approach showcases a competitive performance when compared to state-of-the-art approaches in the contrastive learning domain. Our method offers stability, improved embeddings, and the utilization of prior knowledge for enhanced unsupervised representation learning in Natural Language Processing applications.