Analysing mixed-mode effects of computer assisted telephone and personal interviews: A case study based on survey data from the international social survey programme module on environment

Abstract

Due to Covid-19 restrictions, surveys often could not be conducted in originally planned face-to-face mode, and switched to online modes or used different mixed-mode designs. A combination of CATI and CAPI was used for the Austrian ISSP survey on Environment 2020/2021 (N=1.261), which in the past had always been conducted face-to-face. Mixed-mode surveys facilitate field access in pandemic times and show potential to reduce non-response and coverage errors (desired selection effect). However, the combination of different modes comes along with a series of risks such as mode-effects causing bias due to measurement effects. From an analytical perspective, the challenge arising is to disentangle selection and measurement effects. Thus, we analyse differences in the factorial structure and response distributions of two social constructs using Bayesian multigroup confirmatory factor analysis and linear regression. These represent institutional trust and the willingness to sacrifice for environmental protection. The findings show support for scalar invariance and therefore the absence of CAPI vs. CATI mode-effects on the factorial structure for both constructs. However, despite adjusting for differences in sample composition we observe a higher average willingness within the CATI sample. Based on these results, we discuss implications for the interpretation of mode effects in mixed mode surveys.