Finite Sample Evaluation of Causal Machine Learning Methods: Guidelines for the Applied Researcher

Abstract

The econometrics literature proposed several new causal machine learning methods (CML) in the past few years. These methods harness the strength of machine learning methods to flexibly model the relationship between the treatment, outcome and confounders, while providing valid inferential statements. Whereas numerous options are available now to the applied economics researcher, there is limited guidance on the most useful methodology for a particular applied setting. In this paper, we perform a comprehensive evaluation of the finite sample performance of recently introduced CML methods from the econometrics literature, under a wide range of data generating processes. We focus our analysis on data features that are relevant for causal inference such as varying degrees of: nonlinearity in the outcome and treatment equations, overlap, percentage of treated, alignment and heterogeneity in the treatment effect. We evaluate the methods that have received the most attention so far from the empirical economics literature: double machine learning, causal forest and the generic machine learning methods. Results on the bias, root mean squared error, coverage rates and interval lengths for the average treatment effect, group average treatment effects and individual treatment effects reveal information on the characteristics of the methods and the data features that affect their performance the most.