Comparing the Robustness of Classical and Deep Learning Techniques for Text Classification

Abstract

Deep learning algorithms achieve exceptional accuracies in various tasks. Despite this success, those models are known to be prone to errors, i.e. low in robustness, due to differences between training and production environment. One might assume that more model complexity translates directly to more robustness. Therefore, we compare simple, classical models (logistic regression, support vector machine) with complex deep learning techniques (convolutional neural networks, transformers) to provide novel insights into the robustness of machine learning systems. In our approach, we assess the robustness by developing and applying three realistic perturbations, mimicking scanning, typing, and speech recognition errors occurring in inputs for text classification tasks. Hence, we performed a thorough study analyzing the impact of different perturbations with variable strengths on character and word level. A noteworthy finding is that algorithms with low complexity can achieve high robustness. Additionally, we demonstrate that augmented training regarding a specific perturbation can strengthen the chosen models' robustness against other perturbations without reducing their accuracy. Our results can impact the selection of machine learning models and provide a guideline on how to examine the robustness of text classification methods for real-world applications. Moreover, our implementation is publicly available, which contributes to the development of more robust machine learning systems.