Further Questioning of the Significance of the Gepants

In a recent paper in Headache, titled “The Emperor’s New Gepants: Are the Effects of the New Oral CGRP Antagonists Clinically Meaningful?” by Tfelt-Hansen and Loder, the statistical and clinical significances of the trials for rimegepant and ubrogepant were called into question – and rightfully so. In their letter, the authors remark that while the gepants have achieved statistical significance in trials, their clinical effect may be marginal. However, the statistical significance of these trials may be worth a second look: In addition to using therapeutic gain and number needed to treat, calculating a fragility index (FI) can further assist in assessing statistical significance by complementing the P value. While a P value of <.05 is the current standard for indicating statistical significance, it may be too simplistic and ignore other factors that can affect significance. A shift of only a few events in 1 group could shift the statistically significant results to nonsignificant – or vice versa. Consider 2 statistically significant trials in which the number of randomized patients differs significantly by a factor of 10. In the smaller trial, statistical significance may hinge on only a few events, without which the P value could be shifted to nonsignificance – the P value when taken at face value is unable to communicate that limitation. The utility of the FI therefore is to identify the number of events required to make that shift. It is also important to identify the number of patients lost to follow up and make a direct comparison to the FI; ie, if the number of patients lost to follow up exceeds the FI, the clinician may question if including a certain number of outcomes could have resulted in a shift in statistical significance. Results that are deemed statistically significant with FI less than the number of patients lost to follow up should be interpreted with caution. The FI’s for some of the rimegepant and ubrogepant trials in Table 1 can help when evaluating the statistical significance of these trials. Take for example the ACHIEVE I trial: when comparing the ubrogepant 50 mg group and placebo, the FI is 11 and the number of patients lost to follow up is 45. Had the data for the patients lost to follow up been included, the results of the trial could have shifted from statistically significant to non-significant. Another example is trial 301 which investigated rimegepant: its FI is 2 for its first primary endpoint, meaning that the statistical significance of this study hinged on just 2 participants in a study with N = 1084. Originally the investigators had randomized 1162 subjects, which may indicate that the number of patients lost to follow up may exceed the FI. Reporting the FI can assist clinicians in drawing the appropriate conclusions when considering statistical and clinical significance by complementing the P value.