Trustworthiness criteria for meta-analyses of randomized controlled studies: OBGYN Journal guidelines

Abstract

Meta-analysis is a quantitative statistical technique used to combine and analyze data from the results of multiple previous independent studies on a particular topic, to derive overall conclusions or effect estimates. In general (but not exclusively), meta-analyses are based on RCTs. The results are often used to develop standard practice or clinical guidelines. However, RCTs may be inaccurate or fabricated, leading to journal withdrawal or retraction. This article aims to expand upon the list of RCT quality criteria for authors of meta-analyses of RCTs, so that low-quality and fabricated studies are excluded from meta-analyses.

The editors in the group were invited to participate in monthly or bimonthly calls regarding trustworthiness in OBGYN publishing, with the aim of preventing publication of untrustworthy science in women's health. Using data from the published literature, including our prior work,^{1, 2} Cochrane guidance,³ the TRACT Checklist,⁴ the author instructions of the various journals, and other publications related to trustworthiness of meta-analyses of RCTs,⁵ criteria for meta-analyses were reviewed, reaching consensus by majority.

By consensus, 21 quality criteria were agreed upon by the editors. The aim is for authors to check and confirm the quality criteria for each identified RCT when carrying out a meta-analysis of RCTs (Tables 1 and 2). These criteria help to identify trustworthy RCTs, and are assigned to two groups: absolute criteria, and ‘other quality’ criteria. ‘Absolute’ trustworthiness criteria are those that, if not met, would suggest non-inclusion in the main results of meta-analyses of RCTs (Table 1). ‘Other quality’ criteria are those that, if not met, would suggest lower quality of RCTs (Table 2). In addition, the meta-analysis should be prospectively registered in the PROSPERO database (or a similar international, publicly accessible database, e.g. INPLASY; Research Registry—Registry of Systematic Review/Meta-Analysis).

The consensus decision was that the abstract and primary analysis of meta-analyses should report only trustworthy, ‘high-quality’ RCTs that meet all of the ‘absolute’ criteria (Table 1). Authors of meta-analyses are encouraged to contact RCT authors for additional information regarding the criteria in Tables 1 and 2 if the details cannot be found in the published manuscript or registered protocol. At a minimum, all co-authors of meta-analyses should confirm at the point of submission that each included article meets the criteria included in Table 1. Individual journals may also ask authors to confirm that each article meets the criteria in Table 2, or may go further and ask authors to complete and submit a checklist for the criteria in Tables 1 and 2 for each article included in the meta-analysis. In general, RCTs that are published as an abstract only seldom report all criteria in Tables 1 and 2, and so would often not be included in meta-analyses. Authors could consider a secondary analysis excluding RCTs that, while meeting all ‘absolute criteria’, do not meet some of the ‘other quality’ criteria. Authors of meta-analyses should also provide all items noted in the Submission Checklist for Meta-analyses of RCTs (Table 3). A risk of bias assessment with tools (e.g. Cochrane) may include assessment of allocation concealment, randomization process, etc. These do not have to be entered twice (e.g. in Tables 1 and 2) if already entered in the Risk of bias assessment tool.

Confirming trustworthiness of RCTs is crucial to the integrity of meta-analyses, and assessment must begin with a thorough examination of the published RCTs being considered for inclusion. RCTs are more rigorous than other types of investigations, precisely because they are controlled. A meta-analysis of RCTs combines those studies and extracts data to produce a pooled estimate, and tests for statistical significance. It is more robust than narrative reviews, and synthesizes the evidence rather than simply providing a review of the literature. Meta-analysis of RCTs should meet the Cochrane criteria adopted by PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses), and include the necessary risk of bias assessment.⁹ Subgroup or sensitivity analysis should be considered when there is a high variability or heterogeneity among studies.

A meta-analysis is useful because it may overturn or amplify results from smaller RCTs. Preregistration is an important aspect in reducing bias, ensuring that the study protocols are registered prospectively—i.e. before the first participant is enrolled; changes to this plan during the course of the study can introduce bias. The International Committee of Medical Journal Editors (ICMJE) recommends mandatory prospective registration. Pre-registration contributes to improving reproducibility of research, prevents duplication of efforts, and reduces the potential for bias. The most commonly used registry is ClinicalTrials.gov. Others are listed in the WHO Registry Network. The registry requires data on 24 mandatory elements of a study when pre-registering. The study should follow CONSORT guidelines and the authors should have agreed to share the data if requested.

Retracted RCTs or low-quality studies have often been included in meta-analyses, including those with fabricated or plagiarized data. Evaluating RCTs against the governance criteria outlined in Table 1 will help to address this issue. Including risk of bias assessment of the included studies is important to evaluate the quality of the studies. Trustworthy meta-analyses clearly describe the methods, search strategy, inclusion and exclusion criteria, data extraction procedures, and statistical analysis. If trustworthiness issues are raised, the RCT data should be shared upon request where available.

Including low-quality articles (Table 2) in a meta-analysis reduces the trustworthiness of the results. Trustworthy meta-analyses mitigate publication bias through techniques such as funnel plot asymmetry, statistical tests, and sensitivity analyses. Factors that contribute to trustworthiness include transparency, allowing evaluation of the thoroughness of the meta-analysis.

Using the criteria suggested in this paper (Tables 1–3), authors can conduct trustworthy meta-analyses that will promote the progression of science with integrity and reliability. The results may then be applied with confidence into standard clinical practice or clinical guidance.

The authors have declared they have no conflicts of interest.