Reviewer recommendations and editorial outcomes in peer review: a longitudinal analysis of agreement and disagreement across review rounds.

Abstract

Background: Peer review is often portrayed as a consensus-driven process in which reviewer recommendations largely determine editorial decisions. However, manuscript-level evidence describing how reviewer disagreement unfolds across review rounds-and how it is resolved in routine editorial practice-remains limited.

Methods: We conducted a meta-research analysis of anonymized editorial data from Healthcare (MDPI, Basel, Switzerland; ISSN 2227-9032) (2021-2025), including manuscripts that were ultimately published and manuscripts rejected after external peer review. Reviewer recommendations (accept, minor revision, major revision, reject) were examined at the manuscript level across review rounds. Alignment between reviewer recommendations and editorial outcomes was assessed primarily using the majority recommendation in the final review round, applying a conservative tie-breaking rule (reject > major > minor > accept). Reviewer discordance was defined as the presence of two or more distinct recommendation categories within a review round. Longitudinal trajectories from first to final round were analyzed, with sensitivity analyses using alternative definitions of unfavorable recommendations. The analysis was conducted using anonymized editorial records. The author did not access identifiable reviewer or author information, and the dataset was analyzed in aggregated form to minimize the risk of re-identification.

Results: The analysis included 12,187 published manuscripts and 3,819 rejected manuscripts, corresponding to 21,497 and 5,189 review-round records, respectively. Reviewer discordance was common in the first round for both published and rejected manuscripts (74.8% and 85.8%). Across rounds, discordance declined markedly among published manuscripts (to 29.3% in the final round) but remained high among rejected manuscripts (71.4%). Overall alignment between the majority recommendation in the final round and the editorial outcome was high (≈88%) but not absolute: 6.3% of published manuscripts had a majority recommendation of rejection in the final round, and 8.0% retained at least one rejection recommendation. Longitudinally, discordance was frequently resolved among published manuscripts (53.1% transitioned from discordant to concordant), whereas it often persisted among rejected manuscripts (61.4% remained discordant).

Conclusions: In routine practice, peer review functions less as a voting mechanism and more as an adjudicative process in which editorial judgment integrates heterogeneous reviewer input. The asymmetric resolution of disagreement across outcomes underscores the central role of editorial decision-making when reviewer recommendations diverge.