Evaluating the validity of web-based reaction-time tasks for assessing L2 grammatical knowledge in young learners

Abstract

This study investigated the validity of web-based psycholinguistic tasks for assessing automatized explicit and implicit grammatical knowledge in young second language (L2) learners. Specifically, it examined whether a time-pressured grammaticality judgment task and a self-paced reading task administered online produce results comparable to those obtained in an in-person setting. In addition, the influence of the Big Five personality traits on task participation and performance was explored. A total of 192 Japanese first-year middle school students were assigned to either a School group (in-class setting) or a Home group (online setting). The results revealed that accuracy performance on both tasks was generally comparable across settings, supporting the feasibility of remote task administration. However, reaction-time-based measures were more sensitive to testing conditions. In the grammaticality judgment task, the expected reaction-time effect (faster responses to grammatical items) was observed in the School group but not in the Home group. Similarly, the Home group produced faster and more variable reading times in the self-paced reading task, and the expected slowdown for grammatical errors was not observed, raising concerns about the validity of online reading-time data. Personality traits, particularly neuroticism, conscientiousness, and agreeableness, were associated with task participation and reaction-time variability, highlighting the role of individual differences in online task behavior. These findings demonstrate the need for caution when using reaction-time measures in unsupervised web-based experiments with young L2 learners and offer practical recommendations for enhancing data quality in remote L2 assessment.