Evaluating Methods to Prevent and Detect Inattentive Respondents in Web Surveys

Abstract

Inattentive respondents pose a substantial threat to data quality in web surveys. We evaluate methods for preventing and detecting inattentive respondents. First, we test the effect of asking respondents to commit to providing high-quality responses at the beginning of the survey on various data quality measures. Second, we compare the proportion of flagged respondents for two versions of an attention check item instructing them to select a specific response versus leaving the item blank. Third, we propose a timestamp-based cluster analysis approach that identifies clusters of respondents who exhibit different speeding behaviors. Our findings show that the commitment pledge had no effect on the data quality measures. Instructing respondents to leave the item blank significantly increased the rate of flagged respondents (by 16.8 percentage points). The timestamp-based clustering approach efficiently identified clusters of likely inattentive respondents. Lastly, we show that inattentive respondents can have substantial impacts on substantive analyses.