Don’t Get Lost in the Crowd: Best Practices for Using Amazon’s Mechanical Turk in Behavioral Research

Abstract

The use of Amazon’s Mechanical Turk (MTurk) to conduct academic research has steadily grown since its inception in 2005. The ability to control every aspect of a study, from sampling to collection, is extremely appealing to researchers. Unfortunately, the additional control offered through MTurk can also lead to poor data quality if researchers are not careful. Despite research on various aspects of data quality, participant compensation, and participant demographics, the academic literature still lacks a practical guide to the effective use of settings and features in MTurk for survey and experimental research. Therefore, the purpose of this tutorial is to provide researchers with a recommended set of best practices to follow before, during, and after collecting data via MTurk to ensure that responses are of the highest possible quality. We also recommend that editors and reviewers place more emphasis on the collection methods employed by researchers, rather than assume that all samples collected using a given online platform are of equal quality. We also recommend that editors and reviewers place more emphasis on the collection methods employed by researchers, rather than assuming that all samples collected using a given online platform are of equal quality.