Cognition-aligned frequency filtering for sentence embeddings

Abstract

Learning better sentence embeddings that capture precise semantic plays an important role in Natural Language Processing (NLP). The Sentence Textual Similarity (STS) of embeddings reflects their semantic precision, as this task requires a direct comparison of semantic meanings in vector space. Thus, we focus on improving the ability of sentence embeddings to capture semantic similarity. From the perspective of human cognition, we identify a critical cognitive gap in frequency-domain semantic representation: while semantic information is distributed across all frequency components of embeddings, the human selective attention mechanism suggests that only specific frequency bands are utilized for semantic processing. This frequency-domain cognitive gap leads to semantic redundancy in machine-learned embeddings, which is particularly detrimental for tasks requiring redundancy-resistant representations. To bridge this gap, we propose a simple Cognition-Aligned Frequency Filtering (CAFF) method for unsupervised embedding training on $10^6$ sentences from Wikipedia. CAFF introduces a self-adaptive Frequency Filtering Unit (FFU) to modulate the frequency components of embedding. The FFU functions as a filtering mechanism that suppresses irrelevant components in embeddings to mitigate semantic redundancy. Extensive evaluations with SentEval show that our embeddings improve over the initial encoder by 2.33% on the STS task, achieving state-of-the-art performance. Additionally, our results demonstrate improved performance on both transfer and retrieval tasks.