Reducing bias in the peer-review process

Abstract

Systemic biases in the scholarly review process erode public trust in science. It is incumbent upon editorial boards to consider policy and procedural changes to mitigate biases where they arise. Many scholars have advocated double-blind review (DBR) to offset reviewer biases, yet literature on the value of DBR versus single-blind reviews has been challenging to interpret. For example, early research purported favorable outcomes for female first-authors under DBR (Budden et al. 2008), but reanalysis of those same data (Webb et al. 2008) and more recent (Cox and Montgomerie 2019) and more robust studies (Fox et al. 2023) have not supported that effect.

Staff at the journal Functional Ecology conducted what I consider to be the most relevant and comprehensive peer journal study on the subject. Over 3 years, starting in 2019, Fox et al. (2023) randomly assigned papers submitted to Functional Ecology to either a single-blind review process (i.e., authors identified to reviewers, reviewers blinded to authors; n = 1,837) or double-blind process (author identities blinded to reviewers, reviewers blinded to authors; n = 1,852). They examined differences between review types in terms of overall reviewer ratings (on a scale of 1-4):

4. An extremely novel paper that is in the top 10% of all papers you have read in the broader field of ecology.

3. A strong contribution to the broader field of ecology.

2. Solid work, but largely confirmatory.

1. Weak or flawed, or not of enough importance and general interest for Functional Ecology.

They also tracked editorial decisions and whether the gender or geography of the lead author affected the manuscripts' success.

Compared to single-blind reviews, Functional Ecology papers receiving DBR yielded a lower rating overall (Fox et al. 2023). These differences translated into single-blinded papers being 24% more likely to be invited for resubmission and 15% more likely to have an overall positive outcome compared to DBR papers. No advantages were observed for DBR with respect to gender of the first author. However, papers having first authors from countries with a lower human development index or lower English proficiency fared worse under single-blind review, while equivalent outcomes were observed across demographic groups under DBR. The authors were careful to explain that this pattern reflected a positive bias under single-blind review towards authors from high-income countries rather than a negative bias towards authors from low-income countries. Papers from low-income countries received similar ratings and success regardless of review type. Although authors from lower-income and Non-English countries are more likely to choose DBR when given the option (McGillivray and De Ranieri 2018), Fox et al. (2023) suggested that unless all papers submitted to the same journal are fully blinded, there probably is no benefit conferred to those who choose it. Following the Fox et al. (2023) study, Functional Ecology transitioned to mandatory double-blind review.

What the Fox et al. (2023) paper revealed has been referred to as prestige, affiliation, reputation, or institution bias (i.e., the Matthew effect; Lee et al. 2013), all of which refer to positive reviewer biases towards authors, institutions, or countries from which they expect high-quality work. There is a well-documented bias in favor of the well-known (Ross et al. 2006, Okike et al. 2016); authors familiar to reviewers either through personal connection or author prominence are more likely to have their papers accepted than are unfamiliar authors (Sandström and Hällsten 2008, Okike et al. 2016, Tomkins et al. 2017). Well-known authors from prestigious universities presumably achieved their status by submitting exceptionally high-quality work, and it would follow that rates of acceptance for submissions out of their labs should be high regardless of whether their names are revealed to reviewers or not (Cox and Montgomerie 2019). Yet, success rates for prominent researchers consistently exhibit a 10-15% drop under DBR (Ross et al. 2006, Okike et al. 2016).

Many ecology-focused journals have decided that the benefits of DBR outweigh the costs. Behavioral Ecology, Conservation Biology, Diversity and Distributions, Ecology and Society, Functional Ecology, Journal of Applied Ecology, and Oikos each require DBR.

Other journals, such as Nature, have gone the other direction and opted for a fully transparent review process under the auspices of Open Science. The review side of Open Science may include, among other things, open identities (where authors are known to reviewers, reviewers are known to authors, but reviews are closed to the public), open reports (authors, reviewers, and their reviews and rebuttals are open to the public; the Nature model), open interaction (direct, reciprocal interaction between authors and reviewers), open final-version commenting (public review or commenting on the final published paper), and open data (code and data used in the paper are shared with reviewers and ultimately published alongside the paper; Ross-Hellauer et al. 2023). Moves toward Open Science are intended to promote greater accountability by making conflicts of interest more apparent, generating more thoughtful critiques, and incentivizing quality reviews, as they can be used to support career advancement (Walsh et al. 2000, Polka et al. 2018, Bravo et al. 2019). To use another sports analogy, fully open reports operate on a skin-in-the-game heuristic principle (Haffar et al. 2019), with meta-analyses of the biomedical literature indicating increased quality of published reports (Bruce et al. 2016, Haffar et al. 2019) and decreasing rates of rejection (Bruce et al. 2016), although evidence overall remains scarce (Ross-Hellauer 2023). Notably, Gerwing et al. (2021) indicated that well-trained editors might achieve similar accountability by redacting or editing clearly biased or unkind reviews and calling out such behaviors. There are also concerns that open identities and open reports may harm the review process because of fear of reprisal (Ross-Hellauer et al. 2023), especially among early career professionals and under-represented communities, which could diminish the diversity of viewpoints (Tennant and Ross-Hallauer 2020) and dilute criticism (Ross-Hellauer et al. 2023). In their cross-disciplinary survey, Ross-Hellauer et al. (2023) observed strong support for open interaction, open reports, and final-version commenting but strong push-back against opening reviewer identities to authors, with the majority believing it would make peer review worse. Although most assessments of open science practices focus on attitudes rather than actual outcomes, attitudes matter, and a potential reduction in the available reviewer pool is not a trivial consideration given the ever-increasing challenge of securing suitable reviewers.

This gives us a framework within which to monitor outcomes pending any changes we might adopt.

The Wildlife Society leadership and editorial team are committed to publishing the highest quality research and ensuring integrity in all aspects of the publishing process. This review has identified areas where we might do better, and by extension, better serve the TWS membership and wildlife profession.

The author declares no conflicts of interest.