Chapter 5 Peer Review and the Production of Scholarly Knowledge: Automated Textual Analysis of Manuscripts Revised for Publication in Administrative Science Quarterly

This chapter extends research on peer review by utilizing and assessing an emerging methodology: automated textual analysis. In a corpus of 38 papers successfully revised for publication in Administrative Science Quarterly, the authors found that measures based on exact wording (measured by plagiarism detection) and sentence similarity (measured by Word Mover’s Distance) performed well in capturing differences between original submissions and published papers. They identified the same overall pattern of revision that authors reported (intensive revision of Theory and Discussion sections, limited modification of Methods), and were strongly correlated with the turnover in references and hypotheses that occurred in the course of peer review. Automated textual analysis can usefully contribute to the study of manuscript change in peer review and other social scientific contexts, particularly as available textual corpora grow in size.