Study outcome measurements: statistically insignificant or non-inferior?

Abstract

I read with interest the article by Sim et al. comparing remimazolam vs. propofol in older patients undergoing gastrectomy [1] and would like to add a few insights.

First, Sim et al. state that ‘Evidence suggesting that remimazolam increases the incidence of delirium in older patients compared with propofol is unavailable’ [1]. However, a meta-analysis based on many previous studies observed that the incidence of delirium was similar to other drugs [2]. Therefore, it is a well-established fact that remimazolam neither increases nor decreases the incidence of delirium significantly. Interestingly, this meta-analysis was cited by Sim et al. [1]. Consequently, the statement, ‘the incidence of delirium in older patients undergoing general anaesthesia with remimazolam was assumed to be non-inferior to that in those receiving general anaesthesia with propofol’ [1] is questionable, suggesting that the concept of the study is flawed.

Second, this was published as a ‘randomised non-inferiority study’ [1]. However, it does not adhere to the protocols of a non-inferiority study [3], right from registration to publication. There was no mention of non-inferiority in the study design in the trial registration. Furthermore, the non-inferiority margin should be calculated depending on a previously published superiority trial. There are many studies published on this topic as per the meta-analysis [2]. Unfortunately, Sim et al. fixed the margin as 10 without considering these points which was a violation of the protocol. In addition, as the figure illustrating the ‘position of the confidence interval in relation to the non-inferiority margin, and null value’ [3] was not provided, it is difficult to draw firm conclusions. This raises the question of whether Sim et al. used non-inferiority in the title purely for effect, and it was done merely because of the insignificant p values.

Third, ‘A non-inferiority trial seeks to determine whether a new treatment is not worse than a reference treatment by more than an acceptable amount’ [3]. In addition, the focus should be on whether ‘the new treatment has some other advantage, such as greater availability, reduced cost, less invasiveness, fewer adverse effects or greater ease of administration’ [3]. The study did not examine pharmaco-economics; notably, remimazolam costs about four times more than propofol.

In summary, a non-inferiority study is more complex and challenging [4] than simple randomised trials. It might not be feasible or rather impossible to apply this study design for most clinical studies in our specialty. Clinicians should be aware of the specific protocols of this type of study [3] and adhere to them.