Simultaneous evaluation of the imprecision and inconsistency domains of GRADE can be performed using prediction intervals

Abstract

Objectives

To explore the use of prediction interval (PI) for the simultaneous evaluation of the imprecision and inconsistency domains of Grading of Recommendations, Assessment, and Evaluation using stakeholder-provided decision thresholds.

Study Design and Setting

We propose transforming the PI of a meta-analysis from a relative risk scale to an absolute risk difference using an appropriate baseline risk. The transformed PI is compared to stakeholder-provided thresholds on an absolute scale. We applied this approach to a large convenience sample of meta-analyses extracted from the Cochrane Database of Systematic Reviews and compared it against the traditional approach of rating imprecision and inconsistency separately using confidence intervals and statistical measures of heterogeneity, respectively. We used empirically derived thresholds following Grading of Recommendations, Assessment, and Evaluation guidance.

Results

The convenience sample consisted of 2516 meta-analyses (median of 7 studies per meta-analysis; interquartile range: 5–11). The main analysis showed the percentage of meta-analyses in which both approaches had the same number of certainty levels rated down was 59%. The PI approach led to more levels of rating down (lower certainty) in 27% and to fewer levels of rating down (higher certainty) in 14%. Multiple sensitivity analyses using different thresholds showed similar results, but the PI approach had particularly increased width with a larger number of included studies and higher I² values.

Conclusion

Using the PI for simultaneous evaluation of imprecision and inconsistency seems feasible and logical but can lead to lower certainty ratings. The PI-based approach requires further testing in future systematic reviews and guidelines using context-specific thresholds and evidence-to-decision criteria.

Plain Language Summary

The prediction interval (PI) addresses both the imprecision and inconsistency domains of certainty. In this study, we applied this PI approach to simultaneously judge both domains and compared this to the traditional approach of making these separate judgments. The 2 approaches had moderate agreement. The PI-based approach requires further testing in future systematic reviews and guidelines using context-specific thresholds and evidence-to-decision criteria.