Too Good to Be True: Bots and Bad Data From Mechanical Turk.

IF 10.5 1区心理学 Q1 PSYCHOLOGY, MULTIDISCIPLINARY

Perspectives on Psychological Science Pub Date : 2024-11-01 Epub Date: 2022-11-07 DOI:10.1177/17456916221120027

Margaret A Webb, June P Tangney

引用次数: 0

Abstract

Psychology is moving increasingly toward digital sources of data, with Amazon's Mechanical Turk (MTurk) at the forefront of that charge. In 2015, up to an estimated 45% of articles published in the top behavioral and social science journals included at least one study conducted on MTurk. In this article, I summarize my own experience with MTurk and how I deduced that my sample was-at best-only 2.6% valid, by my estimate. I share these results as a warning and call for caution. Recently, I conducted an online study via Amazon's MTurk, eager and excited to collect my own data for the first time as a doctoral student. What resulted has prompted me to write this as a warning: it is indeed too good to be true. This is a summary of how I determined that, at best, I had gathered valid data from 14 human beings-2.6% of my participant sample (N = 529).

查看原文本刊更多论文

好得不真实：来自 Mechanical Turk 的机器人和不良数据》（Too Good to Be True: Bots and Bad Data from Mechanical Turk.

心理学正越来越多地转向数字数据源，亚马逊的 Mechanical Turk（MTurk）就是其中的佼佼者。2015 年，在顶级行为和社会科学期刊上发表的文章中，估计有高达 45% 的文章包含了至少一项在 MTurk 上进行的研究。在这篇文章中，我总结了自己在MTurk上的经验，以及我如何推断出我的样本--根据我的估计，最多只有2.6%是有效的。我分享这些结果是为了警示和呼吁大家谨慎行事。最近，我通过亚马逊的 MTurk 开展了一项在线研究，作为一名博士生，我第一次渴望并兴奋地收集自己的数据。结果促使我写下这篇文章以示警告：这的确好得不像真的。本文总结了我是如何确定我最多只收集到了 14 个人的有效数据--占我的参与者样本（N = 529）的 2.6%。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Perspectives on Psychological Science PSYCHOLOGY, MULTIDISCIPLINARY-

CiteScore

22.70

自引率

4.00%

发文量

111

期刊介绍： Perspectives on Psychological Science is a journal that publishes a diverse range of articles and reports in the field of psychology. The journal includes broad integrative reviews, overviews of research programs, meta-analyses, theoretical statements, book reviews, and articles on various topics such as the philosophy of science and opinion pieces about major issues in the field. It also features autobiographical reflections of senior members of the field, occasional humorous essays and sketches, and even has a section for invited and submitted articles. The impact of the journal can be seen through the reverberation of a 2009 article on correlative analyses commonly used in neuroimaging studies, which still influences the field. Additionally, a recent special issue of Perspectives, featuring prominent researchers discussing the "Next Big Questions in Psychology," is shaping the future trajectory of the discipline. Perspectives on Psychological Science provides metrics that showcase the performance of the journal. However, the Association for Psychological Science, of which the journal is a signatory of DORA, recommends against using journal-based metrics for assessing individual scientist contributions, such as for hiring, promotion, or funding decisions. Therefore, the metrics provided by Perspectives on Psychological Science should only be used by those interested in evaluating the journal itself.