Risk of Bias When Using Early Failure Criteria in Randomized Clinical Trials With Stereoacuity Outcomes.

Abstract

Purpose: The purpose of this study was to explore the effects of early failure criteria for participants in randomized clinical trials (RCTs) on overall trial conclusions.

Method: We simulated 10,000 hypothetical RCTs with 2 treatments, 1 linear improvement and 1 with increasing rate of improvement and 6 follow-up visits. Each RCT had 400 participants, with the same baseline stereoacuity distribution. We incorporated random test-retest noise for every visit, and scores were rounded to the nearest observable score. Early failure was defined as worsening of two or more levels. We compared mean outcome stereoacuity between treatment groups, with and without the failure rule, using the two-sample t-test and the proportion of erroneous RCTs (significantly different mean outcome values, where truth is known to be no different). Sensitivity analyses were performed to explore the influence of sample size, baseline distribution of stereoacuity, overall magnitude of mean improvement, magnitude of change for the failure rule, and distribution of noise.

Results: A greater proportion of 10,000 simulated RCTs had an erroneous mean difference in outcome with the early failure rule than without (5.49%, 95% confidence interval [CI] = 5.05% to 5.94% vs. 0, 0%, 95% CI = 0% to 0.000001%, difference 5.49%, P < 0.0001). Sensitivity analysis revealed that increased sample size and wider distribution of noise had the greatest influence on increasing proportions of erroneous RCT conclusions.

Conclusions: Study designs incorporating participant-level early failure rules increase the risk of erroneous RCT conclusions and should be avoided.

Translational relevance: We provide data informing the design of future clinical trials. Earlier failure rules at the participant level should be avoided.