Allie Spencer Patterson , Christopher Nicklin , Joseph P. Vitta
{"title":"Methodological recommendations for webcam-based eye tracking: A scoping review","authors":"Allie Spencer Patterson , Christopher Nicklin , Joseph P. Vitta","doi":"10.1016/j.rmal.2025.100244","DOIUrl":"10.1016/j.rmal.2025.100244","url":null,"abstract":"<div><div>Internet-based experiment administration has enabled remote access to diverse populations and large pools of crowdsourced participants. In recent years, webcam-based eye tracking has extended these benefits to a research paradigm historically limited to researchers with sufficient resources for access to technology. While a proliferation of eye-tracking research could reap benefits for applied linguistics and related fields, reporting standards should be modified and updated to account for not only the unique logistic flexibility offered via online administration, but also the methodological limitations of the technology, which include lower spatial/temporal sensitivity and stringent calibration requirements. To facilitate the creation of such standards, a methodological scoping review of 31 experiments published in 22 studies was conducted. Researchers were found to have adhered to psycholinguistic reporting standards for sample size and participant demographics. However, reporting of minimum inclusion criteria, specifically for minimum hardware standards, was often found to lack sufficient detail for replication and requires refinement. Furthermore, critical experiment details, such as calibration procedures, were often found to lack vital details. Based on the findings derived from this review, we present a list of methodological recommendations for implementing and reporting psycholinguistic webcam-based tracking experiments.</div></div>","PeriodicalId":101075,"journal":{"name":"Research Methods in Applied Linguistics","volume":"4 3","pages":"Article 100244"},"PeriodicalIF":0.0,"publicationDate":"2025-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144711720","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Evaluating the validity of web-based reaction-time tasks for assessing L2 grammatical knowledge in young learners","authors":"Takeshi Ishihara , Akira Hamada","doi":"10.1016/j.rmal.2025.100228","DOIUrl":"10.1016/j.rmal.2025.100228","url":null,"abstract":"<div><div>This study investigated the validity of web-based psycholinguistic tasks for assessing automatized explicit and implicit grammatical knowledge in young second language (L2) learners. Specifically, it examined whether a time-pressured grammaticality judgment task and a self-paced reading task administered online produce results comparable to those obtained in an in-person setting. In addition, the influence of the Big Five personality traits on task participation and performance was explored. A total of 192 Japanese first-year middle school students were assigned to either a School group (in-class setting) or a Home group (online setting). The results revealed that accuracy performance on both tasks was generally comparable across settings, supporting the feasibility of remote task administration. However, reaction-time-based measures were more sensitive to testing conditions. In the grammaticality judgment task, the expected reaction-time effect (faster responses to grammatical items) was observed in the School group but not in the Home group. Similarly, the Home group produced faster and more variable reading times in the self-paced reading task, and the expected slowdown for grammatical errors was not observed, raising concerns about the validity of online reading-time data. Personality traits, particularly neuroticism, conscientiousness, and agreeableness, were associated with task participation and reaction-time variability, highlighting the role of individual differences in online task behavior. These findings demonstrate the need for caution when using reaction-time measures in unsupervised web-based experiments with young L2 learners and offer practical recommendations for enhancing data quality in remote L2 assessment.</div></div>","PeriodicalId":101075,"journal":{"name":"Research Methods in Applied Linguistics","volume":"4 3","pages":"Article 100228"},"PeriodicalIF":0.0,"publicationDate":"2025-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144330206","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"EFL students’ engagement in machine translation-assisted writing: Scale development and validation","authors":"Mariko Yuasa , Osamu Takeuchi","doi":"10.1016/j.rmal.2025.100260","DOIUrl":"10.1016/j.rmal.2025.100260","url":null,"abstract":"<div><div>The use of machine translation (MT) in second language (L2) writing has been criticised for potentially disengaging low-proficiency English as a foreign language (EFL) learners from the writing process. However, little is known about how these learners use MT, let alone how their engagement with the writing process should be empirically measured. Therefore, this study developed and validated a multidimensional scale to assess low-proficiency EFL students’ engagement in MT-assisted writing. A 24-item instrument encompassing behavioural, cognitive, affective, social, and agentic engagement subscales was administered to 773 Japanese university students at the CEFR-A2 level (final sample, <em>N</em> = 708). Exploratory factor analysis with half the participants (<em>n</em> = 354) identified five engagement constructs, with cognitive engagement subdivided into pre-editing and post-editing. Agentic engagement was excluded due to a low factor loading. The hypothesised five-factor model was then tested on the remaining participants (<em>n</em> = 354) through confirmatory factor analysis, which yielded satisfactory reliability, as well as construct, convergent, and discriminant validity. This newly developed scale is a valuable tool for investigating how low-proficiency EFL students engage in L2 writing with MT, which often remains unobservable in classrooms where MT use is discouraged or prohibited.</div></div>","PeriodicalId":101075,"journal":{"name":"Research Methods in Applied Linguistics","volume":"4 3","pages":"Article 100260"},"PeriodicalIF":0.0,"publicationDate":"2025-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145048798","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Generating synthetic data for CALL research with GenAI: A proof-of-concept study","authors":"Dennis Foung , Lucas Kohnke","doi":"10.1016/j.rmal.2025.100248","DOIUrl":"10.1016/j.rmal.2025.100248","url":null,"abstract":"<div><div>Popular tools like ChatGPT have placed generative artificial intelligence (GenAI) in the spotlight in recent years. One use of GenAI tools is to generate simulated data—or synthetic data—when the full scope of the required microdata is unavailable. Despite suggestions for educational researchers to use synthetic data, little (if any) computer-assisted language learning (CALL) research has used synthetic data thus far. This study addresses this research gap by exploring the possibility of using synthetic datasets in CALL. The publicly available dataset resembles a typical study with a small sample size (<em>n</em> = 55) performed using a CALL platform. Two synthetic datasets are generated from the original datasets using the <em>synthpop</em> package and generative adversarial networks (GAN) in <em>R</em> (via the <em>RGAN</em> package), which are both common synthetic data generation methods. This study evaluates the synthetic datasets by (a) comparing the distribution between the synthetic and original datasets, (b) examining the model parameters of the rebuilt linear models using the synthetic and original datasets, and (c) examining the privacy disclosure metrics. The results suggest that <em>synthpop</em> better represents the original data and preserves privacy. Notably, the GAN-generated dataset does not produce satisfactory results. This demonstrates GAN’s key challenges alongside the potential benefits of generating synthetic data with <em>synthpop</em>.</div></div>","PeriodicalId":101075,"journal":{"name":"Research Methods in Applied Linguistics","volume":"4 3","pages":"Article 100248"},"PeriodicalIF":0.0,"publicationDate":"2025-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144895543","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Conducting sociolinguistic interviews via generative AI: A methods tutorial","authors":"Annita Stell, Hao Tran, Peter Crosthwaite","doi":"10.1016/j.rmal.2025.100223","DOIUrl":"10.1016/j.rmal.2025.100223","url":null,"abstract":"<div><div>Sociolinguistic interviews (Labov, 1984) are integral to dialectology studies, providing insights into language variation and the social contexts influencing the emergence of new dialects (Hoffman, 2013; Pizarro Pedraza, 2016). Such data, while incredibly valuable, is typically time-consuming and expensive to collect. However, with the advent of generative AI (GenAI) applications e.g., ChatGPT 4o purported to produce discourse in multiple languages, its affordances for generating sociolinguistic interview data across different dialects of dialect-rich languages remain largely unknown. Building on a Gen-AI-assisted Mandarin/ Vietnamese dialect study in Tran and Stell (2024), this tutorial offers step-by-step guidance on conducting sociolinguistic interviews with ChatGPT. Key steps include generating prompts by establishing a dialogue context, formulating appropriate structured open-ended questions to elicit target dialectal varieties, and a cross-validation process with actual dialect speakers. While acknowledging the potential limitations of conducting sociolinguistic interviews with ChatGPT, this tutorial serves to aid those new to GenAI for dialectology research while suggesting future directions for refining this process as GenAI continues to develop.</div></div>","PeriodicalId":101075,"journal":{"name":"Research Methods in Applied Linguistics","volume":"4 3","pages":"Article 100223"},"PeriodicalIF":0.0,"publicationDate":"2025-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144253234","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Predicting the CEFR level of English listening texts with machine learning methods","authors":"Christopher Robert Cooper","doi":"10.1016/j.rmal.2025.100234","DOIUrl":"10.1016/j.rmal.2025.100234","url":null,"abstract":"<div><div>Comprehension in listening texts is often judged by lexical coverage. However, this might not be easily interpretable for language teachers. The CEFR is becoming increasingly influential due to its standardized descriptors across languages. Learners are often placed into classes based on proficiency level, therefore a CEFR level is likely more interpretable than lexical coverage when judging listening text difficulty. Machine learning methods have been used to predict the CEFR level of English reading texts and learner writing, but no such studies exist for listening. The current study hopes to bridge this gap by investigating the potential to predict the CEFR level of listening texts. A corpus of CEFR-labelled listening texts (728 texts, 345,104 words) was compiled for text classification. Three types of variables were created from the corpus data to evaluate comparative predictive accuracy. The first method used linguistic and acoustic features. The others used text embeddings, which represent semantic meaning. The data was split into four classes: A1, A2, B1, and B2+. The accuracy of each method was evaluated by comparing the predicted label in the test data with the label from the original text. The most accurate method used OpenAI embeddings and Support Vector Machines. The overall accuracy was 0.81, with macro averages of precision = 0.75, recall = 0.78, and f-score = 0.76, indicating balanced classification performance across CEFR levels. This method has the potential to predict the CEFR level of listening texts, which could help practitioners and researchers match learners and participants to appropriate listening texts.</div></div>","PeriodicalId":101075,"journal":{"name":"Research Methods in Applied Linguistics","volume":"4 3","pages":"Article 100234"},"PeriodicalIF":0.0,"publicationDate":"2025-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144518520","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Designing and validating an AI-supported tool for enhancing critical inquiry in EFL education","authors":"Wan Yee Winsy Lai , Paul Kim , Ju Seong Lee","doi":"10.1016/j.rmal.2025.100266","DOIUrl":"10.1016/j.rmal.2025.100266","url":null,"abstract":"<div><div>As Generative AI (GenAI) technologies advance rapidly, educational settings face an urgent need for targeted interventions to cultivate learners’ critical, higher-order inquiry skills, so they can effectively navigate, assess, and apply AI-generated content. The urgency of this imperative is magnified for EFL learners in test-driven educational contexts that foster passive learning behaviors, discourage questioning, and inhibit critical thinking. To address these issues, we developed an AI-powered tool designed to evaluate questions based on Bloom’s Taxonomy, a six-level framework of cognitive processes, ranging from basic recall questions (Level 1) to advanced questions that trigger creative and evaluative thinking (Level 5). In study 1, the reliability of the tool was confirmed through multiple inter-rater tests with strong agreement. In study 2, we implemented an intervention program that integrated Bloom’s Taxonomy, targeted readings, group discussions, and sharing to enhance inquiry skills among EFL undergraduate students. Four statistical analyses in SPSS 29.0—including ICC for inter-rater reliability, Pearson correlation, and regression—were conducted to validate the AI-powered inquiry evaluation tool. Across 174 questions, students’ average inquiry level improved from 3.3 to 4.1 (on a five-level scale), showing a significant 0.8-level increase and meaningful enhancement in question quality. The study provides solid evidence of the reliability and validity of the AI-powered inquiry evaluation tool as an objective, real-time method that enhances the efficiency, consistency, and scalability of assessments, offering valuable guidance for EFL practitioners, curriculum designers, researchers, educators, and institutions in integrating evidence-based, inquiry-driven tools into EFL programs.</div></div>","PeriodicalId":101075,"journal":{"name":"Research Methods in Applied Linguistics","volume":"4 3","pages":"Article 100266"},"PeriodicalIF":0.0,"publicationDate":"2025-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145219400","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"From lab to web: Replicating cross-language translation priming asymmetry in an online environment","authors":"Zhiyi Wu, Mireia Toda Cosi","doi":"10.1016/j.rmal.2025.100247","DOIUrl":"10.1016/j.rmal.2025.100247","url":null,"abstract":"<div><div>In second language (L2) acquisition research, understanding how learners process words across languages is crucial, with the translation priming paradigm consistently revealing that an L2 word can be processed significantly faster after a brief presentation of its translation equivalent in one’s first language (L1) but not vice versa. This study attempted to replicate Chen et al.’s (2014) investigation of translation priming asymmetry with Chinese-English bilinguals in an online environment using the Naodao crowdsourcing platform. We conducted three masked priming lexical decision experiments: two testing L1-to-L2 and L2-to-L1 priming with a 50-ms prime duration, and one examining L2-to-L1 priming with an extended 250-ms prime duration. Results showed that the classic asymmetry pattern was not fully reproducible in this online setting at 50-ms prime duration, with null effects in both directions. However, significant priming effects emerged with the extended prime presentation in the L2-to-L1 direction. These findings suggest that online implementation of timing-sensitive paradigms may require methodological adaptations.</div></div>","PeriodicalId":101075,"journal":{"name":"Research Methods in Applied Linguistics","volume":"4 3","pages":"Article 100247"},"PeriodicalIF":0.0,"publicationDate":"2025-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144723188","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Network Analysis for the investigation of rater effects in language assessment: A comparison of ChatGPT vs human raters","authors":"Iasonas Lamprianou","doi":"10.1016/j.rmal.2025.100205","DOIUrl":"10.1016/j.rmal.2025.100205","url":null,"abstract":"<div><div>A recent study by Yamashita (2024) showcases the usefulness of Many-Facet Rasch Model (MFRM) for the analysis of rater effects within the context of Automated Essay Scoring (AES). Building upon Yamashita's work, we break new ground by using Network Analysis (NA) to interrogate the same dataset comparing ChatGPT and human raters for the evaluation of 136 essays. We replicate the analysis of the original study and show a near-perfect agreement between the results of NA and MFRM. We extend the original study by providing strong evidence of halo effect in the data (including the ChatGPT ratings) and propose two new statistics to assess the consistency of raters. We also present simulation studies to show that the NA estimation algorithm is robust, even with small and sparse datasets. Finally, we provide practical guidelines for researchers seeking to use NA with their own datasets. We argue that NA can complement established methodologies, such as the MFRM, but can also be used independently, leveraging its strong visual representations. Relevant algorithms and R code are provided in the Online Appendix to support researchers and practitioners in replicating our findings.</div></div>","PeriodicalId":101075,"journal":{"name":"Research Methods in Applied Linguistics","volume":"4 2","pages":"Article 100205"},"PeriodicalIF":0.0,"publicationDate":"2025-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143816307","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Assigning CEFR-J levels to English learners’ writing: An approach using lexical metrics and generative AI","authors":"Satoru Uchida , Masashi Negishi","doi":"10.1016/j.rmal.2025.100199","DOIUrl":"10.1016/j.rmal.2025.100199","url":null,"abstract":"<div><div>This study presents the CEFR-based Writing Level Analyzer (CWLA), a novel automated system for assessing English learners’ writing proficiency. It assesses proficiency according to CEFR-J levels, a finely graded adaptation of the CEFR framework tailored to the English as a Foreign Language context, particularly in Japan. CWLA generates scores by combining vocabulary scores with AI-based analytical scores, allowing for a sophisticated, regression-based alignment with CEFR-J proficiency levels. By leveraging both straightforward lexical metrics and advanced AI scoring, CWLA provides accurate and interpretable assessments accessible via a user-friendly web interface. To evaluate CWLA's effectiveness, we conducted validation using the ICNALE GRA dataset, which showed a strong correlation of 0.88 between CWLA scores and human ratings. Additionally, entropy analysis indicated that CWLA's scoring distribution closely resembles human rater patterns, capturing the variability expected in human assessments. Three CEFR/CEFR-J specialists also performed expert validation, resulting in an agreement rate of 83.33 %, thereby supporting the system's alignment with expert judgment. These results suggest that CWLA is a reliable system for detailed CEFR-J level assessment, offering promising applications in language education and learner assessment across proficiency levels.</div></div>","PeriodicalId":101075,"journal":{"name":"Research Methods in Applied Linguistics","volume":"4 2","pages":"Article 100199"},"PeriodicalIF":0.0,"publicationDate":"2025-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143644269","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}