Incorrect blinding assessments are common in meta-analyses published in high impact journals

Guidelines and best practices emphasize the importance of blinding to mitigate potential biases in clinical studies.¹ Randomized controlled trials are considered the be the highest form of evidence and typically the goal has been to conduct a placebo controlled double-blinded trial.² Risk of bias (RoB) assessment is a crucial part of evidence synthesis.³ Typically, this evaluation is carried out using tools such as Cochrane's original RoB tool and the revised RoB 2.0 tool.^{4, 5} The original RoB tool explicitly assesses blinding through separate domains for blinding of participants and personnel (performance bias), as well as blinding of outcome assessment (detection bias).⁴ In RoB 2.0, blinding assessment is part of the measurement of the outcome domain.⁵ However, both tools consider blinding within the context of the outcome's nature, whether it is objective for the patient, treating personnel, the outcome assessor, all of these, or none.

The objectivity of the outcome plays a pivotal role in assessing bias risk when evaluating blinding, especially concerning subjective outcomes where the absence of blinding could significantly affect effect sizes.⁶ Previous meta-epidemiological studies have mostly reported that the blinding has influence only on subjective outcomes.^{6, 7} Interestingly, one of the latest and largest meta-epidemiological studies did not find evidence that blinding would even have an influence on subjective outcomes neither.⁸ Thus, it is plausible to assume that the effect of blinding on objective outcomes is either negligible or even nonexistent in medicine.

A key part of the evidence synthesis is to appraise the evidence certainty. The most widely used method is to assess the evidence certainty according to the Grading and Recommendations (GRADE).⁹ GRADE rates each outcome according to different domains, of which one is risk of bias.¹⁰ Thus, to make an accurate judgment on evidence certainty, risk of bias in studies should be adequately assessed. A previous meta-epidemiological study of neonatal meta-analyses showed evidence certainties were often downgraded due to lack of blinding even in the objective outcomes.¹¹ Thus, inadequate downgrading of evidence certainty may have direct influence to treatment guidelines and patient care.¹² The aim of this study was to investigate how the authors of systematic reviews and meta-analyses published in top level journals and Cochrane have estimated the risk of bias due to blinding and have they noted the objectivity of the outcomes in their assessments.

We conducted a systematic meta-epidemiological review of systematic reviews and meta-analyses. As the RoB 2.0 tool was published in 2019, we searched for intervention meta-analyses published in top five general medicine journals (The Lancet, JAMA, BMJ, Annals of Internal Medicine, and PLoS Medicine, ranked based on the 2022 Journal Impact Factors) published from 2020 to November 2023. Journals were selected based on the Journal Impact Factors, as these may be associated with better review quality.¹³ NEJM and Nature Medicine were excluded as they did not have intervention meta-analyses published during this time-period. For these meta-analyses we searched the 100 most recent Cochrane intervention reviews as a control, as we hypothesized that Cochrane reviews would have the highest adherence to their own guidelines. We included meta-analyses in which the intervention was targeted for an individual regardless of the type of the intervention or type of the control(s) used.

Two authors performed the screening process in the Covidence software and disagreements were solved by discussion between these two. Two authors decided independently whether the risk of bias assessment was properly conducted or not. The third author opinion was asked in cases of disagreement. After two authors had piloted the extraction form, one author performed the data extraction (details could be accessed via contacting the author).

Our main outcome was the proper assessment of blinding. We considered the blinding assessment properly done, when the authors mentioned whether the outcome was subjective or objective, and the risk of bias assessment was performed per outcome and not per study or the authors clearly stated that the outcomes were all considered for example subjective and thus the single risk of bias assessment of each study is sufficient. If the authors did not comment on the outcome objectivity and subjectivity in any way and presented a singly risk of bias assessment for each study, we ranked the risk of bias inadequately performed.

We performed the assessment systematically for all the included studies in the following way to have comprehensive view of the blinding assessment. First, we read the methods and how the authors commented on the risk of bias assessment and which tool they had used. Second, we read the results on how the authors commented the risk of bias findings and what were the sources of bias and whether those affected all outcomes or only some of the outcomes. Third, we examined the risk of bias figures and whether there was only a single risk of bias figure, which represented all the studies, or whether there was risk of bias figures separately for objective and subjective outcomes. Fourth, we checked the supplementary materials for additional files and figures and comments on the risk of bias assessments. Finally, we read the summary of findings tables (if presented) and considered there how the authors commented on the risk of bias findings. We did not assess the correctness of other RoB domains.

We have reported this review according to meta-epidemiological adaptation of the PRISMA (Preferred Reporting Items in Systematic Reviews and Meta-analyses) guideline.¹⁴ We compared the proper assessment of blinding between non-Cochrane and Cochrane journals and between the use of RoB 1.0 and 2.0 tools by calculating proportional difference with 95 % confidence intervals (CI) and p-values were calculated by χ² test.

Our initial search retrieved a total of 301 meta-analyses from non-Cochrane journals and after abstract screening and full text assessments a total of 77 were included for this study. From the Cochrane registry, we assessed a total of 150 latest reviews before 100 were eligible for inclusion (Figure S1). Table 1 shows journal and publication years of the included meta-analyses. Non-Cochrane reviews were less likely to include both subjective and objective outcomes in the same review (55% vs. 74%; Table 1). The use of the revised RoB 2.0 tool was notably more common in non-Cochrane reviews (66% vs. 14%; Table 1).

The review authors stated to have assessed and considered and presented the risk of bias assessment for each outcome specifically in 20 (26%) of the non-Cochrane reviews and 33 (33%) of the Cochrane reviews (difference 6%, CI: −6% to 20%, p = 0.31). Use of the revised RoB 2.0 tool was associated with higher proportion of correct assessment of outcome specific risk of bias, as 31 (48%) were correct in RoB 2.0 and 22 (20%) in RoB 1.0 groups (difference 28%, CI: 14% to 42%, p < 0.001).

We found a worrying rate of properly performed risk of bias assessments in this meta-epidemiological review. The assessment of the risk of bias constitutes the sole ground on the level of evidence determination and thereby, establishing treatment guidelines. As the review guidance suggests the authors to consider leaving high risk of bias studies out of analyses or perform sensitivity analyses with these excluded. Furthermore, risk of bias is part of the evidence certainty assessment, when performed according to the GRADE (Grading of Recommendations, Assessment, Development, and Evaluations) framework.^{10, 15} Downgrading of evidence certainty due to inappropriate risk of bias assessment may have direct effect on the treatment provided to the patients.

The risk of bias 2.0 tool was associated to higher rates of correct assessments. The revised risk of bias 2.0 provides a clear guidance document for the assessment. The guidance states that one should consider whether the knowledge of the intervention can affect the outcome assessment, and this should be performed for each outcome separately. As the tool was developed by the Cochrane it is surprising that in our sample the Cochrane reviews had a notably lower rate of 2.0 tool use than the non-Cochrane journals. A previous meta-epidemiological study have found that the change from the original RoB tool to RoB 2.0 tool is still underway.¹⁶ The proportion of RoB 2.0 tool use was 5% in published Cochrane reviews and 42% in protocols in 2022.

In conclusion, we found that meta-analyses published in top level medical journals had an alarming rate of incorrectly performed risk of bias assessments, meaning that the authors had generalized the risk of bias for the whole study, instead of the desired outcome specific assessment. The use of risk of bias 2.0 tool was associated to a notably higher proportion of correct assessments and thus it should be the recommended risk of bias tool.

The authors declare no conflicts of interest.