A data-driven optimization approach for automated reviewer assignment using natural language processing

Abstract

In many settings, such as project or publication selection, expert reviewers play a pivotal role, as their assessments serve as the primary basis for determining a project's prospective value. The effectiveness of matching and assigning qualified experts to evaluate project proposals can substantially influence the quality of the selection process and, consequently, impact the funding organization's return on investment. Despite its importance, many funding organizations continue to rely on basic manual methods for assigning reviewers. This simplistic approach can compromise the quality of project selection and lead to suboptimal financial outcomes. Moreover, it may hinder the equitable distribution of review workloads and increase conflicts of interest between reviewers and applicants. Consequently, there is a pressing need for a systematic and automated method to enhance the reviewer assignment process.

In this study, we propose an optimization-based approach using natural language processing to automate the reviewer assignment process for project proposals. The proposed approach follows a structured three-stage methodology. First, a comprehensive database is constructed by collecting multilingual data on both proposals and reviewers. Second, word embedding techniques are used to represent texts as vectors, enabling the use of cosine similarity to quantify the relevance between each proposal and reviewer. Reviewer expertise and past evaluation performance are also analyzed using predefined knowledge rules. In the final stage, a multi-objective integer linear programming model assigns reviewers by optimizing proposal-reviewer similarity and reviewer competency while preventing conflicts of interest. Additionally, a max-min strategy is employed to ensure fair treatment of less-advantaged proposals, and two supplementary models are introduced to balance reviewer workloads. Experimental results on a real-world dataset from a regional development agency demonstrate that the proposed system significantly outperforms traditional manual assignment methods. We show that automated reviewer assignment prevents subjective judgements, together with reductions in time and cost of the assignment process.