Are results from randomized trials in anesthesiology robust or fragile? An analysis using the fragility index.

Abstract

AIM In anesthesiology, the findings from randomized controlled trials often underpin guidelines influencing clinical decision-making and therefore directly affect patient care. The aim of this study is to evaluate the fragility index and fragility quotient of randomized controlled trials published in the eight highest ranked anesthesiology journals. In addition, we assess the extent to which risk of bias scores, loss to follow-up, Web of Science Citation Index, and journal impact factor influence fragility index and fragility quotient. METHODS We included randomized trials published between 2014 and 2016 from the eight highest ranking anesthesiology journals based on Clarivate Analytics' Science Citation Index and Google Scholar Metrics: Anesthesiology subcategory. We included journals that published general anesthesia topics and omitted specialty anesthesia journals. The fragility index and fragility quotient for all included trials were calculated. Risk of bias for each trial was evaluated using the Cochrane 'risk of bias' Tool 2.0. RESULTS One hundred and thirty one randomized control trials were included in this analysis. The median fragility index was 3 (interquartile range 1.0-5.5) with a fragility quotient of 0.03 (interquartile range 0.01-0.08). In 11% (14/131) of trials, the number of patients lost to follow-up was greater than the corresponding fragility index. Weak correlations were found between fragility index and total sample size (r = 0.13) and between fragility index and event frequency (r = 0.19). A near-negligible correlation was found between 5-year impact factor and fragility index (r = -0.03) and, similarly, between fragility index and Science Citation Index (r = -0.05). Ten trials were at high risk of bias with the randomization process found to be the domain at the highest risk of bias. CONCLUSION In assessing the fragility of randomized controlled trials published in the top eight anesthesiology journals, our study suggests that statistically significant results in these journals are disconcertingly fragile. The median fragility index calculated from our 131 primary studies reveals that only three nonevents must be replaced with events to negate statistical significance. Although a current scale does not exist for fragility index ranges, many trials published by the top journals in anesthesiology are based on concerning methodology and highly fragile outcomes. With small median sample sizes and few patient events characterizing a large number of these trials, many of today's current guidelines and clinical practices may be founded on research containing statistical significance but lacking clinical significance.