Validation of online L2 vocabulary tests: Test performance across laboratory, virtual meeting, and crowdsourcing contexts

Abstract

Online data collection has become increasingly common in diverse fields, including marketing and psychology, and is gaining ground in applied linguistics. Although concerns have been raised about the validity and reliability of online assessments, previous research on online data collection suggests that, with appropriate precautions, data quality can be comparable to that obtained using in-person methods. However, the validity and reliability of online vocabulary tests have not been thoroughly investigated. To fill this gap, the present study compared the results of online vocabulary tests with those of face-to-face administration. In this study, 159 Japanese university students took the Vocabulary Size Test and Phrasal Vocabulary Size Test in three environments: (a) in-person (laboratory), (b) online with supervision (virtual meeting), and (c) online without supervision (crowdsourcing). Reliability and validity were analysed, and results showed that test performance was largely comparable: test environment and presence or absence of supervision had minimal effects on three out of the four tests, with only the meaning recall format of the Vocabulary Size Test showing significantly inflated scores in the crowdsourcing condition. While the findings suggest that pooling data online and aggregating data from different environments are feasible for vocabulary testing research, they also highlight the need for careful planning in research design to achieve a desirable environment for the participants to take the tests.