Improving the Statistical Power and Reliability of Research Using Amazon Mechanical Turk

Abstract

Amazon Mechanical Turk (MTurk) is an increasingly popular source of experimental participants due to its convenience and low cost (relative to traditional laboratories). However, MTurk presents challenges related to statistical power and reliability. These challenges are not unique to MTurk, but are more prevalent than in research conducted with other participant pools. In this paper I discuss several reasons why research conducted with MTurk may face additional power and reliability challenges. I then present suggestions for dealing with these challenges, taking advantage of the comparative strengths of MTurk. The discussion should be of interest to PhD students and other researchers considering using MTurk or other online platforms as a source of experimental participants as well as to reviewers and editors who are considering quality control standards for research conducted with this participant pool.