Insufficient Statistical Power of the Chi-Square Model Fit Test for the Exclusion Assumption of the Instrumental Variable Method

Regression estimates are biased when potential confounders are omitted or when there are other similar risks to validity. The instrumental variable (IV) method can be used instead to obtain less biased estimates or to strengthen causal inferences. One key assumption critical to the validity of the IV method is the exclusion assumption, which requires instruments to be correlated with the outcome variable only through endogenous predictors. The chi-square test of model fit is widely used as a diagnostic test for this assumption. Previous simulation studies assessed the power of this diagnostic test only in situations with strong violations of the exclusion assumption. However, low to moderate levels of assumption violation are not uncommon in reality, especially when the exclusion assumption is violated indirectly. In this study, we showed through Monte Carlo simulations that the chi-square model fit test suffered from a severe lack of power (< 30%) to detect violations of the exclusion assumption when the level of violation was of typical size, and the IV causal inferences were severely inaccurate and misleading in this case. We thus advise using the IV method with caution unless there is a chance for thorough assumption diagnostics, like in meta-analyses or experiments.