Comparing Amazon’s MTurk and a Sona Student Sample: A Test of Data Quality Using Attention and Manipulation Checks

Abstract

The need for cost-effective data collection leads researchers to explore options, especially for respondent-administered online surveys. Student samples are convenient and cheap for social scientists when students fit the target population. However, student samples are criticized for their homogeneity and lack of generalizability (Kees et al. 2017). Another low-cost option is Amazon Mechanical Turk (MTurk), a crowdsourcing platform used for collecting online data from a seemingly broader population. Despite the appeal, it is important to compare data quality. The purpose here is to compare data quality between MTurk and student samples. To control data quality, researchers rely on several tactics such as screener questions to exclude unqualified respondents (Arndt et al. 2022). Subjective manipulation check and attention-check are used to examine respondent engagement and performance. Completion speed might also indicate effort/attention. Since samples should collect data from participants resembling the target population, sample diversity also serves as an indicator of data quality in this study (Kees et al. 2017; Roulin 2015). However, it should be noted that having a diverse sample does not always guarantee higher sample quality, especially when conducting studies on a homogeneous population.