The Editor's Problem

Abstract

An editor maximizing quality relies on the qualitative recommendation of a reviewer. The reviewer may be biased to accept or reject independently of quality. Using the minimum principle, an averaging rule is found to best reduce the noise introduced by bias: under this rule, an unbiased reviewer accepts when quality is greater than the average quality selected by the editor. For distributions with heavy tails, the probability of acceptance due to bias is bounded away from zero even if almost all reviewers are unbiased. Standards adopted by the editor may be excessive relative to the social optimum. Environments with multiple reviewers, reviewer histories, detailed reviews and competition between editors do not solve the problem and may worsen it. A dynamic peer review demonstrates the inherent fragility of equilibria with informative reviews. The model applies to many settings, including grant review, evidence selection, medical testing, juries and project selection.