Association Between Logical Reasoning Ability and Quality of Relevance Judgments in Crowdsourcing

Abstract

Human assessors are in charge of creating relevance judgments set in a typical test collection. Nevertheless, this approach is not that efficient as it is expensive and time consuming while scales deficiently. Crowdsourcing as a recent technique for data acquisition is a low-cost and fast method for building relevance judgments. One of the most important issues for using crowdsourcing instead of human expert assessors is the quality of crowdsourcing in building relevance judgments. In order to assess this issue, factors that may have significant effects on the quality of crowdsourcing relevance judgments should be identified. The main objective of this study is to find out whether cognitive characteristics of crowdsourced workers significantly associated with quality of crowdsourced judgments, and to evaluate the effect(s) that each of those characteristics may have on judgment quality, as compared with the gold standard dataset (i.e. human assessment). Thus, the judgments of the crowdsourced workers is compared to that of a human judgment, as the overlap between relevance assessments, and by comparing the system effectiveness evaluation provided by human judgment and from worker assessors. In this study, we assess the effects of the cognitive ability namely logical reasoning ability on quality of relevance judgment. The experiment shows that logical reasoning ability of individuals is remarkably correlated with quality of relevance judgments.