A Note on Increases in Inattentive Online Survey-Takers Since 2020

Abstract

Lucid, a popular source of online convenience survey samples, has seen a significant increase in inattentive respondents since 2020. Inattentive participants – respondents who incorrectly answer directed query attention check questions – may be introducing substantial measurement error and attenuation bias. Using data from 152,967 survey respondents across multiple studies conducted between January 2020 and June 2021, we find that inattentive respondents report less reliable demographic data, less stable responses, and are systematically different from attentive respondents. We find some evidence of attenuation bias and mixed evidence that data quality has decreased slightly since 2020 even after filtering for inattentive respondents. We conclude that researchers using Lucid should report if they screened on attentiveness and consider replicating any null results. Such an unexpected increase in inattentiveness in a widely-used platform suggests that future researchers relying on online convenience survey samples should continuously assess data quality.