Searching for the policy-relevant treatment effect in Medicare's ACO evaluations.

Objectives: To explain key challenges to evaluating Center for Medicare and Medicaid Innovation (CMMI) accountable care organization (ACO) models and ways to address those challenges.

Study design: We enumerate the challenges, beginning with the conception of the alternative payment model and extending through the decision to scale up the model should the initial evaluation suggest that the model is successful. The challenges include churn at the provider and ACO levels, beneficiary leakage and spillover, participation in prior payment models, and determinants of shared savings and penalties.

Methods: We explain challenges posed in evaluations of voluntary ACO models vs models in which ACOs are randomly assigned to the treatment group. We also note the relationship between the design used in an evaluation and subsequent plans for scaling up successful models.

Results: The optimal research design is inextricably tied to the plans for scaling up a successful model. Decisions regarding churn, leakage, spillover, and participating in past payment models can alter the estimated effects of the intervention on participants in the model.

Conclusions: If CMMI intends to offer the model to a larger, but similar, group of volunteers, then the estimated treatment effect based on voluntary participants may be the most policy-relevant parameter. However, if the scaled-up population has different characteristics than the evaluation sample, perhaps due to mandatory participation, then the evaluator will need to employ pseudo-randomization appropriate for observational data.