The Burden for High-Quality Online Data Collection Lies With Researchers, Not Recruitment Platforms

IF 10.5 1区心理学 Q1 PSYCHOLOGY, MULTIDISCIPLINARY

Perspectives on Psychological Science Pub Date : 2024-04-22 DOI:10.1177/17456916241242734

Christine Cuskley, Justin Sulik

{"title":"The Burden for High-Quality Online Data Collection Lies With Researchers, Not Recruitment Platforms","authors":"Christine Cuskley, Justin Sulik","doi":"10.1177/17456916241242734","DOIUrl":null,"url":null,"abstract":"A recent article in Perspectives on Psychological Science (Webb & Tangney, 2022) reported a study in which just 2.6% of participants recruited on Amazon’s Mechanical Turk (MTurk) were deemed “valid.” The authors highlighted some well-established limitations of MTurk, but their central claims—that MTurk is “too good to be true” and that it captured “only 14 human beings . . . [out of] N = 529”—are radically misleading, yet have been repeated widely. This commentary aims to (a) correct the record (i.e., by showing that Webb and Tangney’s approach to data collection led to unusually low data quality) and (b) offer a shift in perspective for running high-quality studies online. Negative attitudes toward MTurk sometimes reflect a fundamental misunderstanding of what the platform offers and how it should be used in research. Beyond pointing to research that details strategies for effective design and recruitment on MTurk, we stress that MTurk is not suitable for every study. Effective use requires specific expertise and design considerations. Like all tools used in research—from advanced hardware to specialist software—the tool itself places constraints on what one should use it for. Ultimately, high-quality data is the responsibility of the researcher, not the crowdsourcing platform.","PeriodicalId":19757,"journal":{"name":"Perspectives on Psychological Science","volume":null,"pages":null},"PeriodicalIF":10.5000,"publicationDate":"2024-04-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Perspectives on Psychological Science","FirstCategoryId":"102","ListUrlMain":"https://doi.org/10.1177/17456916241242734","RegionNum":1,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"PSYCHOLOGY, MULTIDISCIPLINARY","Score":null,"Total":0}

引用次数: 0

Abstract

A recent article in Perspectives on Psychological Science (Webb & Tangney, 2022) reported a study in which just 2.6% of participants recruited on Amazon’s Mechanical Turk (MTurk) were deemed “valid.” The authors highlighted some well-established limitations of MTurk, but their central claims—that MTurk is “too good to be true” and that it captured “only 14 human beings . . . [out of] N = 529”—are radically misleading, yet have been repeated widely. This commentary aims to (a) correct the record (i.e., by showing that Webb and Tangney’s approach to data collection led to unusually low data quality) and (b) offer a shift in perspective for running high-quality studies online. Negative attitudes toward MTurk sometimes reflect a fundamental misunderstanding of what the platform offers and how it should be used in research. Beyond pointing to research that details strategies for effective design and recruitment on MTurk, we stress that MTurk is not suitable for every study. Effective use requires specific expertise and design considerations. Like all tools used in research—from advanced hardware to specialist software—the tool itself places constraints on what one should use it for. Ultimately, high-quality data is the responsibility of the researcher, not the crowdsourcing platform.

查看原文本刊更多论文

高质量在线数据收集的责任在于研究人员，而非招聘平台

最近，《心理科学展望》（Webb & Tangney, 2022）上的一篇文章报道了一项研究，在亚马逊的机械土耳其人（MTurk）上招募的参与者中，只有 2.6% 被认为是 "有效的"。作者强调了 MTurk 一些公认的局限性，但他们的核心观点是 MTurk "好得不像真的"，而且它 "只捕获了 14 个人类......"。[out of] N = 529"--具有极大的误导性，但却被广泛重复。本评论旨在：（a）纠正记录（即说明 Webb 和 Tangney 的数据收集方法导致了异常低的数据质量）；（b）为开展高质量的在线研究提供一个视角转变。对 MTurk 的负面态度有时反映出人们对该平台的功能以及在研究中如何使用该平台存在根本误解。除了指出详细介绍 MTurk 上有效设计和招募策略的研究之外，我们还强调，MTurk 并不适合每项研究。有效的使用需要特定的专业知识和设计考虑。就像研究中使用的所有工具一样，从先进的硬件到专业的软件，工具本身也对我们应该如何使用它造成了限制。最终，高质量的数据是研究人员的责任，而不是众包平台的责任。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Perspectives on Psychological Science PSYCHOLOGY, MULTIDISCIPLINARY-

CiteScore

22.70

自引率

4.00%

发文量

111

期刊介绍： Perspectives on Psychological Science is a journal that publishes a diverse range of articles and reports in the field of psychology. The journal includes broad integrative reviews, overviews of research programs, meta-analyses, theoretical statements, book reviews, and articles on various topics such as the philosophy of science and opinion pieces about major issues in the field. It also features autobiographical reflections of senior members of the field, occasional humorous essays and sketches, and even has a section for invited and submitted articles. The impact of the journal can be seen through the reverberation of a 2009 article on correlative analyses commonly used in neuroimaging studies, which still influences the field. Additionally, a recent special issue of Perspectives, featuring prominent researchers discussing the "Next Big Questions in Psychology," is shaping the future trajectory of the discipline. Perspectives on Psychological Science provides metrics that showcase the performance of the journal. However, the Association for Psychological Science, of which the journal is a signatory of DORA, recommends against using journal-based metrics for assessing individual scientist contributions, such as for hiring, promotion, or funding decisions. Therefore, the metrics provided by Perspectives on Psychological Science should only be used by those interested in evaluating the journal itself.