Evaluating Different Covariate Balancing Methods: A Monte Carlo Simulation

Abstract We investigate the effectiveness of five weighting and matching techniques, including propensity score matching (PSM), in improving covariate balance and reducing bias when estimating treatment effects in finite-sample situations through Monte Carlo simulations. King and Nielsen (2019. “Why Propensity Scores Should Not Be Used for Matching.” Political Analysis 27 (4): 435–54) argue that pruning observations based on PSM with 1-to-1 greedy matching can worsen rather than improve covariate balance and can increase the bias in the estimates of treatment effects. In our simulations, we observed this phenomenon not only in PSM with 1-to-1 greedy matching but also in other covariate balancing techniques that King and Nielsen (2019. “Why Propensity Scores Should Not Be Used for Matching.” Political Analysis 27 (4): 435–54) recommend as better matching methods, i.e. Mahalanobis distance matching (MDM) and coarsened exact matching (CEM). Regardless of the weighting/matching techniques and the data generation processes in this study, our findings indicate that matching and weighting under extreme caliper or cut-point settings does not improve covariate balance. In addition, once a substantially improved covariate balance is achieved in a given sample, the estimated bias tends to worsen slightly as the covariate balance continues to improve. Moreover, our simulation results suggest that OLS with proper covariates reduces selection bias as well as other weighting and matching methods. The results suggest that when analyzing observational data, it is important to avoid looking for a one-size-fits-all estimator and to identify the appropriate nonexperimental estimator carefully for the sample by thoroughly investigating the available data’s characteristics.