What Types of Survey Questions are Prone to Interviewer Effects? Evidence Based on 29,000 Intra-Interviewer Correlations From 28 Countries of the European Social Survey

Abstract

Interviewer effects are a common challenge in face-to-face surveys. Understanding the conditions that make interviewer variance more likely to occur is essential in tackling sources of bias. Earlier evidence suggests that certain features of the survey instrument provide more ground for interviewer influence. For instance, attitudinal, sensitive, complex or open-ended questions invite more interviewer variance. In this article, we aim to validate earlier results, previously derived from single-country studies, by using the large cross-national sample of the European Social Survey (ESS). We compare 29,330 intra-interviewer correlations derived from 984 survey questions from 28 countries using data from 10 waves of the ESS. The questions were manually coded based on several characteristics. These features of survey questions were then used as predictors of intraclass correlations (ICCs) in multilevel models. The results show that question characteristics account for a significant portion of the variation in ICCs, with certain types, such as attitude and non-factual questions, items appearing later in the survey, and those using showcards, being especially susceptible to interviewer effects. Our findings have important implications for both interviewer training and questionnaire design.