Too Many Bots: A Lesson for Online Quantitative Data Collection

Abstract

“Bots,” computer software capable of taking surveys for an operator, pose a serious threat to the integrity of research that relies on publicly available online surveys. This paper addresses the issue of bot responses to online surveys and suggests several strategies for reducing and addressing these fraudulent responses. To combat this threat, researchers should employ specific methods for building, distributing, and processing surveys that deter and eliminate bot responses from the dataset. Methods for anti-bot survey design include building bot detection software into the survey, creating trap questions, and writing questions that require specific free-form answers. Survey distribution methods that avoid or hide monetary incentives, use a password-protected link, or employ some other form of population targeting will also receive fewer bot responses. Finally, data should be screened for bots after collection using a set of reliable criteria to identify and remove bot responses.