Have we controlled properly? Problems with and recommendations for the use of control variables in information systems research

Abstract

Statistical controls can ensure accurate estimates of causal effects in the evaluation of alternative explanations. However, the research method literature has raised concerns about the appropriate use of control variables (CVs). In this paper, we propose guidelines for the appropriate use of CVs in IS research. We review the use of CVs in statistical control articles published in MIS Quarterly, Information Systems Research, Journal of Management Information Systems, and Journal of the Association for Information Systems between 2015 and 2019. We review a total of 298 articles and closely examine 72 of them. On average, the articles used 5.63 CVs; 65.3% of the articles did not provide a rationale for their choice of CVs, 58.3% did not report the reliability and validity of their CVs, and none included CVs in their hypotheses. To remedy this situation, we discuss an article that exemplifies the proper use of CVs in IS research and make six recommendations for the proper use of CVs. For IS researchers, this paper advances the understanding of the proper use and reporting of CVs. For IS journal editors and reviewers, it provides recommendations for evaluating the use of CVs in empirical IS research. Ultimately, the proper use of CVs strengthens causal arguments and may even improve the generalizability of findings.