Development and application of a comprehensive glossary for the identification of statistical and methodological concepts in peer review reports

Abstract

The assessment of problems identified by peer researchers during peer review is difficult because the content of these reports is typically confidential. The current study sought to construct and apply a glossary for the identification of methodological and statistical concepts mentioned in peer review reports. Three assessors created a list of 1,036 different terms in 19 categories. The glossary was tested on the confidential PEERE database, a sample of 496,928 peer review reports from various scientific disciplines. The most frequently mentioned terms were related to data presentation (found in 40.3 % of the reports) and parametric descriptive statistics (33.3 %). Review reports suggesting a rejection were more likely to mention methodological issues, whereas statistical issues were raised more frequently in review reports recommending revisions. Across disciplines, methodological issues were more frequently mentioned in social sciences (64.1 %), while health and medical sciences were more predictive for the identification of statistical issues (40.1 %). Female reviewers identified more statistical issues compared to male reviewers. These results indicate that the glossary could be used as an additional tool for the assessment of the content of peer review reports and for understanding what help authors may need in writing research articles.