Assessing the replication landscape in experimental linguistics

Abstract

Replications are an integral part of cumulative experimental science. Yet many scientific disciplines do not replicate enough because novel confirmatory findings are valued over direct replications. To provide a systematic assessment of the replication landscape in experimental linguistics, the present study estimated replication rates for over 50.000 articles across 98 journals. We used automatic string matching using the Web of Science combined with in-depth manual inspections of 210 papers. The median rate of mentioning the search string "replicat*" was as low as 1.6%. Subsequent manual analyses revealed that only eight of these were direct replications, i.e. studies that arrive at the same scientific conclusions as an initial study by using exactly the same methodology. Moreover, only 1 in 1600 experimental linguistic studies reports a direct replication performed by independent researchers. We conclude that, similar to neighboring disciplines, experimental linguistics does not replicate enough.