An exploration of referees’ comments published in open peer review journals: The characteristics of review language and the association between review scrutiny and citations

IF 2.9 4区管理学 Q1 INFORMATION SCIENCE & LIBRARY SCIENCE

Research Evaluation Pub Date : 2021-03-13 DOI:10.1093/RESEVAL/RVAB005

D. Wolfram, Peiling Wang, Fuad Abuzahra

{"title":"An exploration of referees’ comments published in open peer review journals: The characteristics of review language and the association between review scrutiny and citations","authors":"D. Wolfram, Peiling Wang, Fuad Abuzahra","doi":"10.1093/RESEVAL/RVAB005","DOIUrl":null,"url":null,"abstract":"\n Journals that adopt open peer review (OPR), where review reports of published articles are publicly available, provide an opportunity to study both review content characteristics and quantitative aspects of the overall review process. This study investigates two areas relevant to the quality assessment of manuscript reviews. First, do journal policies for reviewers to identify themselves influence how reviewers evaluate the merits of a manuscript based on the relative frequency of hedging terms and research-related terms appearing in their reviews? Second, is there an association between the number of reviews/reviewers and the manuscript’s research impact once published as measured by citations? We selected reviews for articles published in 17 OPR journals from 2017 to 2018 to examine the incidence of reviewers’ uses of hedging terms and research-related terms. The results suggest that there was little difference in the relative use of hedging term usage regardless of whether reviewers were required to identify themselves or if this was optional, indicating that the use of hedging in review contents was not influenced by journal requirements for reviewers to identify themselves. There was a larger difference observed for research-related terminology. We compared the total number of reviews for a manuscript, rounds of revisions, and the number of reviewers with the number of Web of Science citations the article received since publication. The findings reveal that scrutiny by more reviewers or conducting more reviews or rounds of review do not result in more impactful papers for most of the journals studied. Implications for peer review practice are discussed.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":" ","pages":""},"PeriodicalIF":2.9000,"publicationDate":"2021-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"9","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Research Evaluation","FirstCategoryId":"91","ListUrlMain":"https://doi.org/10.1093/RESEVAL/RVAB005","RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"INFORMATION SCIENCE & LIBRARY SCIENCE","Score":null,"Total":0}

引用次数: 9

Abstract

Journals that adopt open peer review (OPR), where review reports of published articles are publicly available, provide an opportunity to study both review content characteristics and quantitative aspects of the overall review process. This study investigates two areas relevant to the quality assessment of manuscript reviews. First, do journal policies for reviewers to identify themselves influence how reviewers evaluate the merits of a manuscript based on the relative frequency of hedging terms and research-related terms appearing in their reviews? Second, is there an association between the number of reviews/reviewers and the manuscript’s research impact once published as measured by citations? We selected reviews for articles published in 17 OPR journals from 2017 to 2018 to examine the incidence of reviewers’ uses of hedging terms and research-related terms. The results suggest that there was little difference in the relative use of hedging term usage regardless of whether reviewers were required to identify themselves or if this was optional, indicating that the use of hedging in review contents was not influenced by journal requirements for reviewers to identify themselves. There was a larger difference observed for research-related terminology. We compared the total number of reviews for a manuscript, rounds of revisions, and the number of reviewers with the number of Web of Science citations the article received since publication. The findings reveal that scrutiny by more reviewers or conducting more reviews or rounds of review do not result in more impactful papers for most of the journals studied. Implications for peer review practice are discussed.

查看原文本刊更多论文

同行评审公开期刊审稿人评论探究：评审语言的特点及评审审查与引用的关系

采用开放同行评审（OPR）的期刊，公开发表文章的评审报告，为研究评审内容特征和整个评审过程的定量方面提供了机会。本研究调查了与稿件评审质量评估相关的两个领域。首先，期刊政策是否会影响审稿人根据评论中出现的对冲术语和研究相关术语的相对频率来评估稿件的优点？其次，以引用量衡量，评论/审稿人的数量与手稿发表后的研究影响之间是否存在关联？我们选择了2017年至2018年在17种OPR期刊上发表的文章的综述，以检查综述者使用对冲术语和研究相关术语的发生率。结果表明，无论审稿人是否被要求表明自己的身份，或者这是否是可选的，套期保值术语用法的相对使用几乎没有差异，这表明在审稿内容中使用套期保值不受期刊要求审稿人表明自己身份的影响。与研究相关的术语存在较大差异。我们将稿件的评论总数、修订轮次和评论人数与文章自发表以来收到的科学网引用次数进行了比较。研究结果表明，对所研究的大多数期刊来说，由更多的评审员进行审查或进行更多的评审或几轮评审并不会产生更具影响力的论文。讨论了对同行评审实践的影响。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Research Evaluation INFORMATION SCIENCE & LIBRARY SCIENCE-

CiteScore

6.00

自引率

18.20%

发文量

期刊介绍： Research Evaluation is a peer-reviewed, international journal. It ranges from the individual research project up to inter-country comparisons of research performance. Research projects, researchers, research centres, and the types of research output are all relevant. It includes public and private sectors, natural and social sciences. The term "evaluation" applies to all stages from priorities and proposals, through the monitoring of on-going projects and programmes, to the use of the results of research.