Peer review reports of randomized controlled trials in oncology can be short and superficial

Objectives

To evaluate the quality of open peer review reports published alongside articles of randomized controlled trials (RCTs) in oncology.

Methods

We searched and sampled from completed parallel RCT articles published in 2021 in 62 BioMed Central journals operating open peer review and evaluated their first-round peer review report. We assessed and described the peer review report content, clarity, and completeness and explored whether reviewers commented on the manuscript's importance, robustness, interpretation, discussion of results, and RCT reporting. Two investigators evaluated the review reports independently, with conflict resolution involving a third author.

Results

We sampled 26 RCTs and evaluated their 59 first peer review reports. Median word count was 276 (range = 0–1047). Only 11 reports were constructive (19%), suggesting solutions for the problems noted. Of reviewers commenting on the manuscript's methods section (n = 46/59, 78%), 74% (n = 34/46) addressed the suitability of the methodology. Fewer commented on the adequacy of conclusions (n = 15/59; 25%) or the applicability of results (n = 5/59; 9%), or whether study limitations had been acknowledged by authors (n = 11/59; 18%). Only four (7%) commented on open research practices, including deviations from protocols, completeness of reporting, and sharing of data and materials.

Conclusion

Peer review reports of published RCTs in oncology were short, superficial, and rarely constructive. Although there is indication that reviewers commented on study methodology, little attention was paid to study conclusions, deviation from study protocols, completeness of reporting, or data availability. Such review reports would be of limited value to authors for improving their trial study manuscripts, or to editors in deciding on manuscript publication.

Plain Language Summary

Clinical trials are research studies that test whether a treatment or action works to help prevent or treat a disease. The results from these studies are important because they help doctors and policy makers decide what care is best for patients. Before the results from a clinical trial are published, other experts, sometimes including members of the public, carefully check the study to make sure it was conducted properly and that the results are trustworthy. This checking process is called “peer review”. Reviewers look at things like how the study was carried out, whether the results make sense, and if the conclusions are fair. In our project, we looked at how well this peer review process worked in a selection of medical journals that make their review reports public. We read the reviews for 26 published trials to see what the reviewers said. We found that most of the review reports were very short and did not provide much detail to help the journal editors decide if the study should be published. Some reviews were only one sentence long. Many reviewers did not comment much on how the study was analyzed, and very few said whether the study conclusions were reasonable. Although peer review is supposed to help make sure research is reliable, we found that sometimes the reviews are too brief to be very helpful.