Patterns of preregistration and publication of trials in Cochrane systematic reviews of interventions

Abstract

Objectives

It is widely recognized that selective reporting clinical trial results based on their outcomes, in the forms of publication bias, outcome reporting bias, or p-hacking, has detrimental effects on the scientific literature and on evidence synthesis. This can be recognized and perhaps ameliorated with comprehensive trial registration. However, previous investigations of clinical trial registration focused on study-level examinations rather than the number of trial participants, which is often more relevant to meta-analysis. Our objective was to investigate the risk of bias from selective reporting considering both trials but also the number of included participants.

Methods

We took a random sample of 50 Cochrane systematic reviews (SRs) of interventions which included randomized controlled trials, forming a retrospective cohort. Focusing on the primary outcome in the SR we used the review, published trial information, and public trial registration documents to collect information about the reviews themselves, as well as information about “included,” “ongoing,” and studies “awaiting classification”.

Results

In all 50 selected reviews, there were 423 “included” trials which examined the primary outcome, of which 109 (25.7%) were preregistered. There was substantial variability in proportions of preregistration of included trials among reviews, with a median of 16.0% (interquartile range 0%−79.6%). Registered trials covered 60.1% of all participants, suggesting larger studies were more likely to be preregistered. The proportion of participants in registered trials which were published was high (98.2%), but the proportion of registered trials which were published also varied substantially between reviews.

Conclusion

We found that in Cochrane reviews, there remains a low rate of preregistration among included studies and evidence for a substantial rate of trial nonreporting of registered trials. However, preregistered trials contributed proportionally more patients to reviews, and findings remain unpublished for only a small proportion of participants in registered trials.

Plain Language Summary

Trials are an important form of evidence in scientific literature that are often combined into systematic reviews (SRs), which give an overview for the evidence in a specific topic. However, trials and therefore the reviews can be misleading when the authors change the outcomes that they report based on the results they find, which is called reporting bias. One way of minimizing reporting bias is to register the planned methods for the trial in advance, known as preregistration. It is known that the proportion of trials that is registered can be low, but when conducting SRs the results are affected more by the number of participants within trials than the number of trials, which has not been previously studied. We aimed to make an up-to-date estimate of the proportion of preregistered trials in Cochrane SRs but also find out the proportion of participants within registered trials in these reviews. To do this we took a random sample of 50 Cochrane SRs, which conduct careful searches for registered trials whether or not they are published, and examined the trials within these reviews that studied the outcome the review declared to be most important. We found that even in modern SRs, only 25% of trials were preregistered, and that this number was very variable in reviews of different clinical questions. However we found that 60% of the patients in the reviews were within the preregistered trials, indicating that preregistered trials are generally larger than unregistered trials. We also found that more than 90% of preregistered studies were published. However, a major problem with this approach is that we cannot detect trials that were started without registration and then never published, which means that our results may underestimate the problem. Overall, this indicates that the risk of reporting bias is somewhat lower when considering participants rather than trials, but the risk of reporting bias is still high even in modern, rigorous reviews in medical science.