Contrastive Transformer Network for Long Tail Classification

Abstract

In the context of big data, multi-label text classification presents considerable challenges, most notably the long-tail problem, wherein a small number of labels account for the majority of instances, while the vast majority of labels occur only rarely. This imbalance creates a critical bias in classification models, leading to suboptimal performance on tail labels that significantly impacts applications such as recommender systems and search engines. We present CTN-LT (Contrastive Transformer Network for Long Tail Classification), a novel dual-encoder architecture that combines adapted loss functions, contrastive learning and reframes the multi-label text classification as a semantic similarity task to specifically enhance tail label performance. Our method achieves state-of-the-art performance on tail labels while maintaining competitive performance on head labels across multiple benchmark datasets. The model demonstrates superior few-shot and zero-shot capabilities, making it particularly valuable for dynamic environments where new categories frequently emerge. We release our code at https://github.com/jmelsbach/CTN-LT.