Evaluating Clinical Implementation of Risk Prediction Based Interventions Using Difference-In-Differences.

Objective: To compare alternative Difference-in-Differences (DID) methods for evaluating the effect of risk-stratified interventions, or interventions targeting at-risk groups, on binary outcomes.

Study setting and design: In simulations, we compared operating characteristics of recycled prediction estimators for common average treatment effect on the treated (ATT) estimands across three DID models: the traditional two groups and two periods model, a risk score adjusted model, and a model adjusting for risk score and its interactions with risk group and period. We compared DID ATT estimates to randomized evaluation estimates of a risk-stratified intervention implemented at Kaiser Permanente Washington (KPWA), delivering additional text-message reminders to reduce missed clinic visits.

Data sources and analytic sample: Our study included 588,503 KPWA visits, with 285,814 (49%) visits pre-evaluation (05/01/2018-10/30/2018) and 302,689 (51%) visits during the evaluation (02/01/2019-09/30/2019). Pre-evaluation, 120,350 visits were classified as high-risk. During the evaluation, 125,076 visits were labeled as high-risk, with 62,557 (50%) randomized to the intervention. We generated data in simulations based on this setting.

Principal findings: In simulations, the traditional DID and risk score adjusted models had smaller bias and standard errors, and better coverage probabilities. DID estimates closest to randomized evaluation estimates (-0.007, 95% CI [-0.010, -0.004]) were from the traditional DID model assuming the identity link (-0.008, 95% CI [-0.011, -0.005]) or the risk adjusted model with any link (-0.006, 95% CI [-0.008, -0.003] identity; -0.007, 95% CI [-0.011, -0.003] logit; -0.007, 95% CI [-0.012, -0.003] log) for the ATT on the absolute difference scale (usual DID ATT estimand), and the risk score adjusted model with log or logit links for all other estimands.

Conclusions: Compared with randomized evaluation results, the traditional DID model is appropriate for the ATT on the absolute difference scale, while the risk score adjusted model with log or logit links is appropriate for all ATT estimands considered.