RevASIDE: Evaluation of Assignments of Suitable Reviewer Sets

Abstract

Scientific publishing heavily relies on the assessment of quality of submitted manuscripts by peer reviewers. Assigning a set of matching reviewers to a submission is a highly complex task which can be performed only by domain experts. We introduce and deeply evaluate RevASIDE, a reviewer recommendation system that assigns suitable sets of complementing reviewers from a predefined candidate pool with- out requiring manually defined reviewer profiles. Here, suitability includes not only reviewers’ expertise, but also their authority in the target domain, their diversity in their areas of expertise and experience, and their interest in the topics of the manuscript. We present three new data sets for the expert search and reviewer set assignment tasks and compare the usefulness of simple text similarity methods to document embeddings for expert search. We analyse the suitability of the approach for different sizes of reviewer sets. Furthermore, a quantitative evaluation demonstrates significantly better results in reviewer set assignment compared to baselines. A qualitative evaluation also shows their superior perceived quality.