GRADE notes 5: comparing a new intervention with an established treatment. Practical and methodological issues

Objectives

Recommendations about the comparison of multiple interventions should ideally be based on direct evidence. Issues may arise in a guideline development group (GDG) if the intervention of interest is compared with an alternative, widely accepted intervention, and direct evidence suggests that the former may be at least as effective as the latter.

Study Design and Setting

In this report, we present our experience during the development of evidence-based recommendations, according to the Grading of Recommendations Assessment, Development and Evaluation methodology, on azathioprine as an off-label treatment for multiple sclerosis in settings with limited resources.

Results

Direct evidence from small studies on critical outcomes (relapse and worsening of disability) probably favored, with very low certainty, azathioprine over interferon (IFN), a widely accepted labeled treatment for multiple sclerosis. Indirect evidence on IFN compared to placebo, despite being supported by larger trials, favored IFN with very low certainty. These observations were surprising for some GDG members and challenged their confidence in the seemingly established health effects of IFN and on how they should comparatively evaluate azathioprine. De facto, the long-established treatment that before the availability of newer disease-modifying treatments had been considered for years as a standard of care, was only based on very low certainty evidence, which was paired by very low certainty evidence in favor of another treatment. To not deviate from the mandate and scope of the guideline, through discussion and voting, conditional recommendations supporting the use of azathioprine were made where IFN and/or other treatments were not available and affordable. Although the GDG concluded that there was insufficient evidence to recommend azathioprine ahead of IFN, some of its members would have preferred to rank azathioprine ahead of IFN, based on the published evidence. An alternative solution may have been to add a new question and rediscuss the role of the established treatment, that is, IFN.

Conclusion

Recommendations should be developed according to the target recommendation question and the scope of the guideline. Scenarios that question the perceived health effects of long-established interventions must be met by an openness to reconsider such standards of care. If directed by the GDG, external, indirect evidence on the widely accepted intervention may have to be assessed and this should be considered when planning a guideline recommendation. Potential alternative approaches on how to formulate the final recommendation should be carefully weighted in relation to the context and setting to which recommendations are targeted.