Benchmarking machine learning methods for the identification of mislabeled data

Abstract

Supervised machine learning recently gained growing importance in various fields of research. To train reliable models, data scientists need credible data, which is not always available. A particularly hard and widespread problem deteriorating the performance of methods are mislabeled samples (Northcutt in J Artif Intell Res 70:1373-1411, 2021). Common sources of mislabeling are weakly defined classes, labels that change their meaning, unsuitable annotators, or ambiguous guidelines for labeling. Because mislabeling lowers prediction quality, it is essential for scientists to be able to identify wrong labels before actually starting the learning process. For that, numerous algorithms for the identification of noisy instances have been developed. However, so far, a comprehensive empirical comparison of available methods has been missing.

In this paper, we survey and benchmark methods for the identification of mislabeled samples in tabular data. We discuss the theoretical background of label noise and how it can lead to mislabeling, review categorizations of identification methods, and briefly introduce 34 specific approaches together with popular data sets. Finally, 20 selected methods are benchmarked using artificially blurred data with controllable mislabeling and a new real-life genomic dataset with known errors. We compare methods varying the amount and the type of noise, as well as the sample size and domain of data. We find that most of the methods have the highest performance on datasets with a noise level of around 20-30% where the best filters identify around 80% of the noisy instances with relatively high precision (0.58\(-\)0.65). Acquiring precise predictions seems to be a more challenging task than identifying most of the noisy instances: while the average recall score over all models ranges from 0.48 to 0.77, the average precision score ranges from 0.16 to 0.55. Furthermore, none of the methods excels over all others in isolation, while ensemble-based methods often outperform individual models. We provide all data sets and analysis code to enable a better handling of mislabeled data and give recommendations on usage of noise filters depending on various dataset parameters.