Impact of a Research Integrity Assessment (RIA) of Randomized Controlled Trials Included in Interventional COVID-19 Systematic Reviews: A Meta-Epidemiological Study

Objective

This study aimed to evaluate the feasibility, reliability, and impact of the Research Integrity Assessment (RIA) tool when applied to randomized controlled trials (RCTs) included in systematic reviews. RIA is a structured tool designed to assess retractions, trial registration, ethical approval, authorship, and plausibility of methods and results, thereby identifying RCTs that may not meet basic standards of research integrity.

Design

Meta-epidemiological study.

Methods

We systematically identified Cochrane reviews and non-Cochrane systematic reviews of RCTs investigating interventions for COVID-19 and extracted all RCTs. Each RCT was independently assessed by two reviewers (with different expertise in evidence synthesis) using the RIA tool, with disagreements resolved by a senior reviewer. Reliability and feasibility were recorded, and sensitivity analyses examined the impact of excluding studies failing the RIA on meta-analytic results.

Results

Two hundred six RCTs from 23 Cochrane reviews and non-Cochrane systematic reviews were assessed with RIA. Fifty-nine RCTs (29%) were excluded due to integrity concerns, 79 (38%) classified as “awaiting classification”, and 11 (5%) identified as non-randomized studies, leaving 57 RCTs (28%) rated as “no concern.” The most common reason for exclusion was absent or retrospective trial registration, while uncertainties around ethics approval were the main reason for “awaiting classification”. Interrater reliability was moderate overall (κ = 0.5), with higher agreement in objective domains and lower in domains requiring interpretive judgment, necessitating senior adjudication in a substantial proportion of assessments. On average, application of RIA required 21–27 min per RCT; however, the time required for senior assessor reassessment, conflict resolution, and author correspondence was not systematically recorded and substantially exceeded that of the initial assessments. We received 35 author responses to 165 individual queries. Sensitivity analyses restricted to RCTs passing RIA reduced the median number of eligible RCTs per meta-analysis by 60%. This frequently widened confidence intervals and decreased the certainty of conclusions, although the direction of effect estimates changed only rarely.

Conclusions

These results demonstrate that integrity checks can be feasibly applied and reveal widespread concerns in COVID-19 RCTs, but require expertise, adjudication mechanisms, and adequate resources. Incorporating such assessments into review methodology can strengthen the trustworthiness of clinical evidence and safeguard patient safety.

Registration

A protocol to this meta-epidemiological study has been registered (https://doi.org/10.17605/OSF.IO/NBRHX).