Judgement and Ranking: Living with Hidden Bias

Abstract

The complexity and subjectivity of the judgement task conceals the existence of biases that undermines the quality of the process. This paper presents a weighted aggregation function that attempts to reduce the influence of biased judgements on the final score. We also discuss a set of desirable properties. The proposed weighted aggregation function is able to correct the "nationalism bias" found by Emerson et al. (2009) in the 2000 Olympic Games diving competition and suggest the possibility of a reputation bias. Our results can be applied to judgement sports and other activities that require the aggregation of several personal evaluations.