What's the point of peer review?

Abstract

If there is one common experience shared by all scientists, regardless of subdiscipline, it is the gauntlet of peer review. We all know the painful experience of rejection, the frustration of acquiescing to reviewers’ demands, and the many months that can sometimes elapse between the submission of and first decision on a paper. But for many, it is the peer-review process that adds the necessary ingredient of rigor—the stamp of approval—to science. For instance, science journalists primarily cover peer-reviewed studies, and the court systems consider peer-reviewed science to be the gold standard in environmental and conservation-related cases.

I have always thought that peer review acted as the primary filter excluding the most egregious error-laden and misguided science from entering the canon of scientific literature. But think about it—how often have you tossed out a paper of yours because it was rejected after peer review? How often have you, after making minimal changes, or no changes at all, re-submitted to another journal hoping for a “better” draw of peer reviewers? Perhaps several decades ago, when all journals were print-only and page space and the number of journal options were limited, the situation really was “make the changes or bust”. But with the remarkable proliferation of journals that now exist in every subdiscipline, every paper can find a home. According to Scopus, there are at least 550 indexed journals in the environmental science subcategory of “ecology”, and that number is growing. After each rejection, you could quite literally re-submit the same paper every few months to a new journal for the rest of your career, and know that you'll get a bite at some point.

The problem of poor-quality science in the literature is worsened by the exponentially growing sector of “predatory” or “pay-to-publish” outlets. These outlets’ journals, which often spam prospective authors with urgent messages asking for a rapid submission, will publish papers with little to no peer-review oversight, and for a fee. Much has been written about this seedy underbelly of academic publishing, and “sting” operations have revealed how little these outlets care about the content in their journals. One of my favorite examples occurred in 2020 when Dr. Dan Baldassarre, a behavioral ecologist at the State University of New York-Oswego, submitted a spoof paper titled “What's the Deal with Birds?” to a suspected predatory journal, the Scientific Journal of Research and Reviews. To the delight of Dr. Baldassarre's followers on social media, the paper was accepted, published within only seven days of its initial submission (!) if the metadata are to be believed, and still stands as one of the greatest publishing punk-jobs in science. Sometimes we have to laugh so that we don't cry; and while this example still makes me chuckle, the problems in publishing do not.

If we cannot trust journals at the “fringe”, then perhaps we can place more trust in science published in the top-tier and society-run journals, such as the one you are reading now. I do think this is a partial solution; professional societies have a strong vested interest in upholding their reputation within the academic community, and society journals are often edited by respected individuals in the field. But we still see retractions of papers, even in very well-regarded journals, and high-profile stories of data fabrication or manipulation have recently come to light. A journal affiliated with a well-known academic society recently discovered a slew of papers published as part of a special issue that were fraudulent. Authors publishing in another society-run journal have expressed concern over the perceived poor quality of peer reviewers’ comments. This hints at the broader problem of an over-burdened population of reviewers. Taken as a whole, though, it is beginning to feel like less and less of what we read in the scientific literature can be trusted.

So, given its shortcomings, what is the point of peer review? While not a silver bullet, peer review does act as a filter, if only a coarse one. And peer review often results in better papers, even if they remain imperfect. Editors-in-Chief, Associate Editors, and reviewers cannot catch all mistakes or erroneous inferences, and sniffing out misconduct is even harder. The limitations of peer review should teach us to bring a healthy skepticism to whatever we read; this eternal questioning is the hallmark of a good scientist, is it not? As the rest of society grapples with how to parse (mis)information on the internet, we too must become more questioning of published science, while avoiding cynicism. Perhaps it is a healthy thing for scientists and society to view peer review less as a stamp of approval, and to become more critical consumers of information.