Population Recruitment Strategies in the Age of Bots: Insights from the What Is on Your Plate Study

Abstract

Background

To evaluate state-wide nutrition policies, valid tools are required to gather sufficient sample sizes. Remote data collection, including web-based dietary assessments, offers convenience for participants and researchers and enables faster and more diverse recruitment. However, it presents challenges, including risk of bots compromising data integrity.

Objectives

This study describes the technical survey design of an ongoing longitudinal study, which is evaluating a state-wide Supplemental Nutrition Assistance Program (SNAP) incentive program, discusses strategies to prevent and identify bots, duplicates, fraudulent entries, and implausible data, and provides recommendations to improve future public health nutrition research.

Methods

From May to September 2023, SNAP participants from Rhode Island and Connecticut were recruited to complete an online food frequency questionnaire (FFQ) and a demographic survey. Given the large sample and online format, our interdisciplinary team designed the technical backend to optimize participants’ convenience while ensuring data quality through an automated system that assessed FFQ responses. To prevent bots and duplicates, we created duplicate application programming interfaces (API), randomly called participants, and evaluated Completely Automated Public Turing Test to Tell Computers and Humans Apart (reCAPTCHA), geotags, and Internet Protocol (IP) addresses.

Results

Using a combination of text blasts and in-person recruitment, we enrolled 1367 participants, with text blasts proving the most effective strategy (∼60% of participants). Midway through recruitment, we identified 544 potential bots that completed the screener, with duplicate IP addresses and geotags from outside the recruitment area serving as strong indicators of bot activity. At baseline, 112 participants failed FFQ data quality checks, prompting follow-up by research assistants. Our automated duplicate and FFQ APIs saved countless hours of staff time.

Conclusions

Remote data collection tools were critical for meeting recruitment goals and ensuring our data authenticity. A combination of strategies is necessary to effectively mitigate against bots and ensure plausible responses. Widely available, built-in tools (e.g., reCAPTCHA) are helpful but are insufficient alone. Customized solutions like our automated systems may be critical for future researchers to maintain data integrity.