Combining a Local Comparison Group, a Pretest Measure, and Rich Covariates: How Well Do They Collectively Reduce Bias in Nonequivalent Comparison Group Designs?

Abstract

This study examined bias reduction in the eight nonequivalent comparison group designs (NECGDs) that result from combining (a) choice of a local versus non-local comparison group, and analytic use or not of (b) a pretest measure of the study outcome and (c) a rich set of other covariates. Bias was estimated as the difference in causal estimate between each NECGD and a carefully appraised randomized experiment with the same intervention, outcome, and estimand. Results indicated that bias generally declined with the number of design elements in an NECGD, that combining all three sufficed to eliminate bias but was not necessary for it, and that this pattern of results was largely replicated across five different replication factors.