Recruitment Strategies Bias Sampling and Shape Replicability.

Abstract

Replicating psychological research has become a central concern for psychologists. Although attention has been paid to the possibility of heterogeneous populations driving replication success/failure, the heterogeneous recruitment strategies researchers use to draw samples from those populations are often overlooked. Yet recruitment strategies may bias the participants who show up and shape replication results. We examine this idea through several unique paradigms (sampling North American university students, N_total = 1,009). First, subtle manipulations of recruitment strategies (i.e., mentioning cash, expedient credit, fun, or a study narrative) were differentially appealing to individuals varying on experiential versus reward-based motivations (Experiment 1). Second, employing different recruitment strategies biased the motivational styles of actual participant show-ups, and sometimes even shaped the success of several replication studies (Experiment 2-3). We conclude that recruitment strategies may sometimes alter the degree of successful replication.