Individual participant data–informed risk of bias assessments for randomized controlled trials in systematic reviews and meta-analyses

Abstract

Objectives

In evidence synthesis, assessing risk of bias (ROB) of eligible studies is crucial to inform interpretation of findings. Standardized tools like Cochrane's ROB-1 or ROB-2 traditionally rely on published information to inform assessments, but this is often incomplete or unclear. Availability of raw individual participant data (IPD) enables more in-depth assessments; however, guidance on how to use IPD in ROB assessments is lacking. We aim to develop preliminary guidance on how to use IPD to inform ROB assessments of randomized controlled trials (RCTs) for three case studies.

Study Design and Setting

In stage 1, we reviewed relevant literature, consulted our networks, and drew on previous experience to compile items on how IPD may inform ROB assessment for each domain. We discussed feasibility and potential usefulness of each item with an international, interdisciplinary expert advisory group and developed preliminary guidance, which was piloted in two IPD meta-analyses (MAs) (65 RCTs) using ROB-1. In stage 2, the guide was adapted for ROB-2 and applied to another IPD-MA (34 RCTs). All assessments were conducted in duplicate by two independent reviewers. In stage 3, we conducted an evaluation workshop to further refine each item, and capture important lessons. To assess the impact of IPD-informed assessments, we compared them to existing ROB-1 assessments performed with published information alone for 33 trials.

Results

We identified 12 items across the ROB domains. IPD provided opportunities to enhance ROB assessments by enabling additional checks for selection bias (ie, testing randomization) and attrition bias (ie, more granular assessment of incomplete data at various time points). We also identified domains for which availability of IPD enabled reduction of ROB, for instance, by mitigating selective outcome reporting bias or by reincluding excluded participants in intention-to-treat analyses. Applying IPD-informed assessments led to changes in ROB judgment in 25 of 33 studies, most commonly, resolution of domains previously marked as “unclear”.

Conclusion

Our preliminary guidance for IPD-informed ROB assessments may be applied in IPD-MAs to increase the accuracy of ROB assessments and in some cases reduce ROB to create a more reliable evidence base informing policy and practice.

Plain Language Summary

When making decisions about how to treat a patient in clinical practice, it is important to consider the results of all relevant studies. Usually, combined analyses of multiple clinical trials rely on published reports, in which researchers summarize their findings. However, looking at the original data from these studies, instead of just the published reports, can improve the quality of analyses. Access to these underlying data also allows for more thorough assessment of the studies' quality and any potential for bias. This is important for understanding the results properly and for making the most appropriate treatment decisions for patients. Here, we present guidance on how to assess risk of bias of trials using these original datasets.