Mechanical Turk Versus Student Samples: Comparisons and Recommendations

Abstract

Mechanical Turk and other online crowdsourcing markets (OCMs) have become a go-to data source across scientific disciplines. In 2014 Steelman and colleagues investigated how Mechanical Turk data compared with student samples and consumer panels. They found the data to be comparable and reliable for academic research. In the nearly 10 years since its publication, the use of Mechanical Turk in research has grown substantially. To understand whether their results still hold, we conducted a partial replication to determine how Mechanical Turk workers continue to compare with students using UTAUT 2 as our theoretical model and virtual-reality headsets as the focal IT artifact. Our findings generally align with Steelman et al. (2014) and confirm that Mechanical Turk continues to offer a suitable alternative to student samples. This study reveals consistent results between the student and OCM samples, indicating the potential for interchangeability. The OCM samples are primarily male, while the student sample is majority female, following current US academic trends. All samples are significantly different in age, and only the US OCM and non-US OCM samples are similar in education. The path coefficients from the non-US OCM sample differ significantly from those from other OCM samples; the path coefficients derived from the student sample do not differ significantly from any OCM sample. While sample differences exist, as expected, many are addressable post hoc if anticipated and designed for during data collection. From our findings and the extant literature, we summarize recommendations for researchers and review teams.