Deep Learning Methods with Pre-Trained Word Embeddings and Pre-Trained Transformers for Extreme Multi-Label Text Classification

Abstract

In recent years, there has been a considerable increase in textual documents online. This increase requires the creation of highly improved machine learning methods to classify text in many different domains. The effectiveness of these machine learning methods depends on the model capacity to understand the complex nature of the unstructured data and the relations of features that exist. Many different machine learning methods were proposed for a long time to solve text classification problems, such as SVM, kNN, and Rocchio classification. These shallow learning methods have achieved doubtless success in many different domains. For big and unstructured data like text, deep learning methods which can learn representations and features from the input data wtihout using any feature extraction methods have shown to be one of the major solutions. In this study, we explore the accuracy of recent recommended deep learning methods for multi-label text classification starting with simple RNN, CNN models to pretrained transformer models. We evaluated these methods’ performances by computing multi-label evaluation metrics and compared the results with the previous studies.