Labeling Agreement Level and Classification Accuracy

Abstract

This paper covers an experiment that investigates the relation between the quality of the dataset and the performance of the classifier. It demonstrates that dataset with less noisy labels, i.e., higher agreement level between labelers can achieve better classification accuracy results. In order to set the experiment, we divided human annotated Arabic Twitter dataset under two levels of majority voting ratio: low and high. Then, we reported the credibility prediction accuracy results under these two levels. It was found that by using labeled dataset with low level of agreement between labelers means low ratio of majority voting class, the accuracy was in the range (32% - 50.5%) whereas with labeled dataset with high percentage of majority voting class, it was between (62.8% - 66.7%). This finding clarifies that improving the quality of labeling by reducing the effect of noisy labels would yield better classification results.