Evaluating the agreement between sensitivity and primary analyses in observational studies using routinely collected healthcare data: a meta-epidemiology study.

Abstract

Background: Sensitivity analysis is a crucial approach to assessing the "robustness" of research findings. Previous reviews have revealed significant concerns regarding the misuse and misinterpretation of sensitivity analyses in observational studies using routinely collected healthcare data (RCD). However, little is known regarding how sensitivity analyses are conducted in real-world observational studies, and to what extent their results and interpretations differ from primary analyses.

Methods: We searched PubMed for observational studies assessing drug treatment effects published between January 2018 and December 2020 in core clinical journals defined by the National Library of Medicine. Information on sensitivity analyses was extracted using standardized, pilot-tested collection forms. We characterized the sensitivity analyses conducted and compared the treatment effects estimated by primary and sensitivity analyses. The association between study characteristics and the agreement of primary and sensitivity analysis results were explored using multivariable logistic regression.

Results: Of the 256 included studies, 152 (59.4%) conducted sensitivity analyses, with a median number of three (IQR: two to six), and 131 (51.2%) reported the results clearly. Of these 131 studies, 71 (54.2%) showed significant differences between the primary and sensitivity analyses, with an average difference in effect size of 24% (95% CI 12% to 35%). Across the 71 studies, 145 sensitivity analyses showed inconsistent results with the primary analyses, including 59 using alternative study definitions, 39 using alternative study designs, and 38 using alternative statistical models. Only nine of the 71 studies discussed the potential impact of these inconsistencies. The remaining 62 either suggested no impact or did not note any differences. Conducting three or more sensitivity analyses, not having a large effect size (0.5-2 for ratio measures, ≤ 3 for standardized difference measures), using blank controls, and publishing in a non-Q1 journal were more likely to exhibit inconsistent results.

Conclusions: Over 40% of observational studies using RCD conduct no sensitivity analyses. Among those that did, the results often differed between the sensitivity and primary analyses; however, these differences are rarely taken into account. The practice of conducting sensitivity analyses and addressing inconsistent results between sensitivity and primary analyses is in urgent need of improvement.