Methodological Techniques to Estimate the Sensitivity of the Agency for Healthcare Research and Quality Patient Safety Indicators.

Background: Little is known about how comprehensively the Agency for Healthcare Research and Quality's patient safety indicators (PSIs) capture true complications. Therefore, the authors sought to assess the PSIs' sensitivity using a novel sampling and analytic strategy tailored for unusual events to ensure adequate capture of false negative cases.

Methods: The authors retrospectively reviewed hospitalization records not flagged by 7 selected PSIs, oversampling those with specific diagnosis or procedure codes suggesting an unreported complication, with a special interest in PSI 09 (Postoperative Hemorrhage or Hematoma) and PSI 10 (Postoperative Physiologic and Metabolic Derangement). The authors evaluated data from 27 hospitals in 11 states between 2006 and 2009. For each PSI, the authors determined the negative predictive value (NPV), accounting for sampling weights, and used previous estimates of positive predictive value (PPV) and incidence to estimate sensitivity.

Results: For PSI 09, 32 of 281 abstracted records (including 30 of 116 high-risk records) were falsely negative (NPV 99.73%; 97.5%, confidence interval [CI], 98.96-99.94); the estimated sensitivity was 40% (95% CI, 12-76). For PSI 10, 3 of 230 records (including 3 of 108 high-risk records) were falsely negative (NPV 99.92%; 97.5% CI, 99.28-99.99); the sensitivity was 53% (95% CI, 9-92). The estimated sensitivity of other PSIs varied (19%-100%).

Conclusions: The sensitivity of several Agency for Healthcare Research and Quality PSIs, estimated from a sample of hospitalizations enriched with records suggesting an unreported complication, varied widely. Although the 2-stage complex stratified sampling design (using weights based on sampling probabilities) allows estimation of the sensitivity of hospital outcome measures, large sample sizes are still required for unusual events.