Group Decision Making as Consistency Measure for Learning To Rank

Abstract

Quality of training data deeply influences Machine Learning performance. Lack of consistency in terms of coherence of labels for similar items can mislead the learning model. This work studies the role of training data consistency for machine learning algorithms dealing with ranking problems, i.e., the Learning to Rank (LTR) methods. In particular, a consistency measure is introduced by leveraging the consensus value broadly adopted in Group Decision Making (GDM). Then, the statistical relationship between the proposed consistency measure and the performance of a deep neural network implementing an LTR method is evaluated. Experimentation confirms the suitability of the proposed measure and reveals a heavy correlation between GDM consensus and deep learning model performance. Such information could drive data filtering at the training stage and guide model update decisions.