Off-task behavior negatively impacts performance.

Abstract

Remote data collection has become more common in psychological research on human participants due to the large, diverse participant pool available at relatively inexpensive costs. This shift, however, comes at the expense of the oversight and carefully controlled setting found in the laboratory. The present study examined the off-task behavior of participants hired through Amazon's Mechanical Turk (MTurk) and its impact on memory performance across two pre-registered experiments. In Experiment 1, we found roughly one third of participants went off task during the encoding portions of a memory study and that this off-task behavior predicted poorer performance on the memory test. In Experiment 2, we replicated this outcome and further demonstrated that an explicit warning to stay on task reduced page switching but did not reduce the proportion of time spent off task. These results suggest the ease of remote data collection comes at a cost: a small but sizable minority of MTurk participants voluntarily navigate away from their study tasks, which negatively impacts performance. Further research is necessary to better understand how researchers might mitigate such behavior.