High robustness does not always imply low uncertainty of treatment rankings: An empirical study of 60 network meta-analyses

Abstract

Background: Network meta-analysis computes treatment ranking to assist with clinical decision making, but it is not always clear how reliable the ranking is and how likely the accumulation of new evidence may alter the ranking. Uncertainty and robustness of ranking are two concepts related to the reliability of ranking. However, it is still unclear whether these two approaches would always yield similar conclusions on the reliability of ranking, i.e., a robust ranking is also one of low uncertainty. Purpose: This study aimed to investigate the relationship between the uncertainty and robustness of treatment ranking by using normalized entropy and quadratic weighted Cohen’s kappa, respectively. Data. We used datasets of previously published NMAs from a database maintained by Petropoulou et al. at the University of Bern. Analysis. Scatter plots and Pearson’s correlation coefficients were used to demonstrate the direction and strength of the association between uncertainty and robustness of ranking for NMA-level and treatment-level evaluation. Results: We found that when the uncertainty of ranking is very low, treatment ranking is unlikely to be altered by deleting a trial from the complete data. However, network meta-analysis with robust treatment ranking may have high uncertainty of treatment ranking. Conclusions: Therefore, although the robustness of the ranking can find the trial that has the most significant impact on the ranking, the high robustness of ranking does not mean that the ranking would not easily change when new trials are added in the future.