Peer Review—Can AI Help?

One of my biggest concerns related to journal editing is the inconsistency in peer review. The fate of a manuscript often seems to depend on who is assigned as AE and who is called on to provide peer review comments. Let me qualify that. A really top-end paper will get accepted no matter what, and a really poor submission will be rejected accordingly. It's the manuscripts that fall in between where there is variability.

In my experience, each editor has a slightly different bar by which to judge a manuscript. First, the scientific editors evaluate each manuscript as assigned. They look for a variety of parameters to judge: new and novel work, robust experimental design with appropriate replications and statistical assessment, clear and concise presentation and discussion, appropriate abstract and conclusions, low similarity index, among others.

Each editor has their own bar regarding grounds for rejection, some much lower than others. This is evident in the rejection rates of individual editors. On average, SEs reject about half of the manuscripts that come to them, the other half being forwarded to AEs to process. But the variation among SEs runs from a low of 3.6% rejected immediately to as high as 84%. That is, one SE essentially sends everything to an AE, while another sends less than 2 out of 10 forward. Although some of this variation may be related to the topic area of interest for each SE, the variation among editors is still high.

Of those manuscripts sent to an AE, on average, about 50% more are rejected, but again, each AE has their own bar. A couple of AEs reject only about 15% of their manuscripts, while a couple of others reject over 80%. One AE rejects virtually everything sent to them.

Then we need to factor in the quality of the peer reviewers who evaluate a manuscript. Again, in my experience, this is widely variable; some reviewers provide 2–3 pages of insightful commentary, while others barely provide a sentence with little to no justification of their decision. We strive to get three good critical reviews for each manuscript, but often we don't reach that goal. AEs get to rate each review, so we know who does a consistently good job and who not to trust as much.

It's these numbers that raise concerns about the equity of our peer review system.

Some have suggested that we use AI to help make the peer review process easier and perhaps more consistent. But there are some huge roadblocks to this practice, and we still do not allow its use in peer review. The main issue is that it is not appropriate to feed a manuscript into a program like ChatGPT, since that breaks the confidentiality of the peer review process itself. And still, can you really trust everything that ChatGPT says?

Some LLMs do not require material to be added to the database, like Copilot. Still, we do not allow Copilot's use to conduct the review. While AI can provide interesting insights, depending on the questions asked, human judgment is still required to differentiate the fine points.

Over the past year or so, we have started asking reviewers to declare any AI usage when they submit their reviews. We recently reviewed their responses to this question. The vast majority (over 95%) said no AI sources were used. Those who declared its use stated it was simply for polishing the grammar of their own review comments; one of these even said they wrote their review in their native language and then used AI to translate to English.

Still, AI is here, and applications are growing rapidly. Why not harness the power of AI to assist in peer review and other aspects of scholarly publishing? This is, in fact, the topic of Peer Review Week, an annual event scheduled for September 15–19 this year. This event, a series of meetings, podcasts, webinars, and blog posts from a range of experts, is intended to “foster collaboration and awareness” on a new topic chosen each year. It's not surprising that the use of AI was chosen as the topic for this year.

The main points for discussion include ethics, literacy and training, fairness and transparency, human and machine judgment to provide a basis for how AI can best be used to enhance peer review. Hopefully, some new guidance about AI use in peer review will emerge from these discussions.

Whether we're ready or not, AI is here. Although reviewers generally do not use AI to assist them with the critical aspects of evaluating science, we need to find ways to optimize its use without breaching confidentiality. And maintain the most important aspect of peer review, human critical thinking.

Sincerely,

Rich Hartel, PhD

Editor in Chief, Journal of Food Science

Professor, University of Wisconsin–Madison