Measurement issues in causal inference

Abstract

Research in the social and behavioral sciences relies on a wide range of experimental and quasi-experimental designs to estimate the causal effects of specific programs, policies, and events. In this paper we highlight measurement issues relevant to evaluating the validity of causal estimation and generalization. These issues impact all four categories of threats to validity previously delineated by Shadish et al. (Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin, Boston, 2002): internal, external, statistical conclusion, and construct validity. We use the context of estimating the effect of the COVID-19 pandemic on student learning in the U.S. to illustrate the important role of measurement in causal inference. We provide background related to the meaning of measurement, and focus attention on the evidence and argumentation necessary to evaluate the validity and reliability of the different types of measures used in statistical models for causal inference. We conclude with recommendations for researchers estimating and generalizing causal effects: provide clear statements for construct interpretations, seek to rule out potential sources of construct-irrelevant variance, quantify and adjust for measurement error, and consider the extent to which interpretations of practical significance are consistent with scale properties of outcome measures.