Reanalyzing Tourangeau (2017) and Brick and Tourangeau (2017): A Research Note on Nonresponse Bias

Abstract

In two important studies, Tourangeau (2017) and Brick and Tourangeau (2017) reanalyzed Groves and Peytcheva’s (2008) meta-analytic data on nonresponse bias. They found that: 1) 23% of the variation in nonresponse bias (absolute relbias) is between studies, and 2) there is a sizable and robust study-level correlation between bias and response rates, which is much larger than the estimate-level correlation (more than twice as large with analytic weights applied). Initially, however, these authors examined two different measures of nonresponse bias, absolute differences and absolute relbias, because of concerns the absolute relbias measure was misleading. Unfortunately, only the absolute relbias results were included in the two published articles. In the current study, I replicate and extend their published results, fixing an error in their analysis. I also provide the omitted results for absolute differences. The published and omitted results lead to dissimilar conclusions. In the omitted findings for absolute differences, much more of the variation (40-46%) in nonresponse bias is at the study level, but the study-level correlation between bias and response rates tends to be much weaker, weaker even than the estimate-level correlation, is sometimes positive, is rarely statistically significant, and is highly sensitive to model specification. The dissimilarities in the results for absolute differences versus absolute relbias raise important new questions for future research.