KDD 2021 Tutorial on Systemic Challenges and Solutions on Bias and Unfairness in Peer Review

Abstract

Introduction. Peer review is a cornerstone of academic practice [1]. The peer review process is highly regarded by the vast majority of researchers and considered by most to be essential to the communication of scholarly research [2–4]. However, there is also an overwhelming desire for improvement [2, 4, 5]. Problems in peer review have consequences much beyond the outcome for a specific paper or grant, particularly due to the widespread prevalence of the Matthew effect (“rich get richer”) in academia [6]. As noted by [7] “an incompetent review may lead to the rejection of the submitted paper, or of the grant application, and the ultimate failure of the career of the author.” (See also [8, 9].) The importance of peer review and the urgent need for improvements, behooves research on principled approaches towards addressing problems in peer review, particularly at scale. In this tutorial, we discuss a number of key challenges in peer review, outline several directions of research on this topic, and also highlight important open problems that we envisage to be exciting to the community. This document summarizes the contents of the tutorial and provides relevant references.