Do data collection methods matter for self-reported L2 individual differences questionnaires? In-person vs crowdsourced data

Abstract

Crowdsourcing offers great advantages in data collection by enabling researchers to recruit a large number of participants across geographical boundaries within a short period of time. Despite the benefits of crowdsourcing, no study has explored its validity in collecting self-reported individual differences (ID) data in second language (L2) research. The present study aims to address this gap by examining crowdsourcing as a viable alternative or complementary tool to traditional in-person data collection. We recruited a total of 209 in-person and 209 crowdsourced participants for comparison. Both groups completed the short versions of the Foreign Language Classroom Anxiety Scale and the Foreign Language Enjoyment Scale, provided their demographic and language learning background information, and completed the LexTALE test. Measurement invariance testing revealed that most (sub)constructs exhibited partial or full invariance, indicating stability in the measurement systems across both data collection settings. However, crowdsourced participants reported higher enjoyment and lower anxiety than in-person participants. These differences can be attributed to the more relaxed mental state of the crowdsourced participants who completed the survey outside of the classroom. Moreover, some crowdsourced participants tended to overrate their English proficiency and exhibited potentially dishonest behavior during the LexTALE test. These findings suggest that although crowdsourcing offers valuable opportunities for data collection in L2 ID research, the potential for inflated self-assessments and questionable behavior in an unsupervised online testing environment must be considered. Thus, the use of crowdsourcing platforms to collect self-reported L2 ID data requires caution and careful preparation.