Research Methods in Applied Linguistics最新文献_第8页

Assigning CEFR-J levels to English learners’ writing: An approach using lexical metrics and generative AI 为英语学习者的写作分配CEFR-J水平：一种使用词汇度量和生成人工智能的方法

Research Methods in Applied Linguistics Pub Date : 2025-03-19 DOI: 10.1016/j.rmal.2025.100199

Satoru Uchida , Masashi Negishi

{"title":"Assigning CEFR-J levels to English learners’ writing: An approach using lexical metrics and generative AI","authors":"Satoru Uchida , Masashi Negishi","doi":"10.1016/j.rmal.2025.100199","DOIUrl":"10.1016/j.rmal.2025.100199","url":null,"abstract":"<div><div>This study presents the CEFR-based Writing Level Analyzer (CWLA), a novel automated system for assessing English learners’ writing proficiency. It assesses proficiency according to CEFR-J levels, a finely graded adaptation of the CEFR framework tailored to the English as a Foreign Language context, particularly in Japan. CWLA generates scores by combining vocabulary scores with AI-based analytical scores, allowing for a sophisticated, regression-based alignment with CEFR-J proficiency levels. By leveraging both straightforward lexical metrics and advanced AI scoring, CWLA provides accurate and interpretable assessments accessible via a user-friendly web interface. To evaluate CWLA's effectiveness, we conducted validation using the ICNALE GRA dataset, which showed a strong correlation of 0.88 between CWLA scores and human ratings. Additionally, entropy analysis indicated that CWLA's scoring distribution closely resembles human rater patterns, capturing the variability expected in human assessments. Three CEFR/CEFR-J specialists also performed expert validation, resulting in an agreement rate of 83.33 %, thereby supporting the system's alignment with expert judgment. These results suggest that CWLA is a reliable system for detailed CEFR-J level assessment, offering promising applications in language education and learner assessment across proficiency levels.</div></div>","PeriodicalId":101075,"journal":{"name":"Research Methods in Applied Linguistics","volume":"4 2","pages":"Article 100199"},"PeriodicalIF":0.0,"publicationDate":"2025-03-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143644269","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Systematic comparison of computational measures of linguistic synchrony in online educational environments 在线教育环境中语言同步计算度量的系统比较

Research Methods in Applied Linguistics Pub Date : 2025-03-13 DOI: 10.1016/j.rmal.2025.100195

Jinnie Shin, A. Pauline Aguinalde

{"title":"Systematic comparison of computational measures of linguistic synchrony in online educational environments","authors":"Jinnie Shin, A. Pauline Aguinalde","doi":"10.1016/j.rmal.2025.100195","DOIUrl":"10.1016/j.rmal.2025.100195","url":null,"abstract":"<div><div>Linguistic synchrony, the alignment of linguistic features between conversational partners, is a key indicator for understanding students’ learning outcomes in computer-mediated learning environments, where communication quality directly influences success. In online tutoring, synchrony fosters better comprehension and engagement. Despite numerous methods to measure synchrony—such as lexical, syntactic, and semantic alignment—systematic comparisons across these approaches remain limited, impeding a full understanding of how synchrony operates across different educational contexts. This study addresses this gap by analyzing tutoring conversations from diverse contexts, particularly English as a Second Language (ESL) tutoring and algebra instruction, using seven computational models. These models yielded 29 synchrony indices, ranging from traditional lexical overlap to advanced natural language processing-based embedding models, capturing synchrony across multiple dimensions. Our results show that while these methods generally align with theoretical dimensions of synchrony, variations in embedding methods and distance measures can significantly impact their ability to capture meaningful interactions. Additionally, different subject domains influenced the finer-grained dimensions of synchrony observed, particularly in temporal aspects.</div></div>","PeriodicalId":101075,"journal":{"name":"Research Methods in Applied Linguistics","volume":"4 2","pages":"Article 100195"},"PeriodicalIF":0.0,"publicationDate":"2025-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143610744","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Centering Pacific knowledge of time, space, and relationality in community-based translanguaging research: An engaged and fluid temporal methodology 以社区为基础的跨语言研究中时间、空间和关系的太平洋知识为中心：一种参与和流动的时间方法论

Research Methods in Applied Linguistics Pub Date : 2025-03-13 DOI: 10.1016/j.rmal.2025.100198

Honiara Amosa , Sam Amosa , Corinne A. Seals

{"title":"Centering Pacific knowledge of time, space, and relationality in community-based translanguaging research: An engaged and fluid temporal methodology","authors":"Honiara Amosa , Sam Amosa , Corinne A. Seals","doi":"10.1016/j.rmal.2025.100198","DOIUrl":"10.1016/j.rmal.2025.100198","url":null,"abstract":"<div><div>In Western perspectives, space and place are often viewed as distinct concepts. Place refers to a specific physical location, whereas space describes the interactions that take place within that location. Time is typically understood as a fixed moment or the interval between moments. In contrast, Pacific perspectives see time, space, and place as interconnected with each other and with individuals. In this worldview, a space or place continues to resonate even after one leaves it, and time is seen as circular rather than linear, linking the past, present, and future. This interconnectedness is embodied in the concept of 'va', which emphasizes the importance of understanding relationships and responsibilities within the community involved in research (cf. Airini et al., 2010). This article argues for the importance of recognizing Indigenous epistemologies and ontologies as valid in their own right, which in this case means recognizing the importance of va in Pacific research. We also present our theorization of an Engaged and Fluid Temporal Methodology, which outlines via five principles how Pacific understandings guided our work with Pacific communities. We then walk through each of the five principles as a methodological guide for community-based research, providing illustrative examples from the Wellington Translanguaging Project. By focusing on principles to research that are imbued with respective Pacific understandings of time, space, and relationships (e.g. va), we argue that this supports the research to remain in and for the communities with which we worked, a core component of Indigenous research methodology.</div></div>","PeriodicalId":101075,"journal":{"name":"Research Methods in Applied Linguistics","volume":"4 1","pages":"Article 100198"},"PeriodicalIF":0.0,"publicationDate":"2025-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143610704","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Assessing Whisper automatic speech recognition and WER scoring for elicited imitation: Steps toward automation 评估Whisper自动语音识别和对诱导模仿的WER评分：迈向自动化的步骤

Research Methods in Applied Linguistics Pub Date : 2025-03-11 DOI: 10.1016/j.rmal.2025.100197

Michael McGuire , Jenifer Larson-Hall

{"title":"Assessing Whisper automatic speech recognition and WER scoring for elicited imitation: Steps toward automation","authors":"Michael McGuire , Jenifer Larson-Hall","doi":"10.1016/j.rmal.2025.100197","DOIUrl":"10.1016/j.rmal.2025.100197","url":null,"abstract":"<div><div>Elicited imitation (EI) has received considerable attention in the field of SLA as a psycholinguistic method for oral proficiency assessment. However, EI tests are perhaps underutilized because of the need for time-consuming scoring by hand. A clear solution to this is computer automation, which requires two key components to work in parallel: (1) an accurate automatic speech recognition (ASR) system to transcribe EI response speech into text, and (2) a computational scoring metric to assess the resulting text. In this paper, we test the feasibility of automating EI scoring with the open-source Whisper ASR system and the Word Error Rate (<em>WER</em>) scoring metric which measures deviation from the original prompt. 30 Japanese L1 participants took a 30-item test, resulting in 900 English L2 learner EI sentence responses which were transcribed by the two authors and Whisper ASR. The intraclass correlation coefficient (ICC) between the error rates of the human raters and Whisper across all items was ICC = 0.929, 95 % CI [0.921, 0.936] indicating very strong alignment. We then compared automated test scores using <em>WER</em> to those done manually by human raters using a traditional ordinal-scale scoring method developed by Ortega et al. (2002) and found a robust correlation of <em>r</em> = 0.969, 95 % CI [0.935, 0.985] across overall participant scores. These findings show that the combination of the Whisper ASR system and the <em>WER</em> scoring metric result in EI test scores that align almost perfectly with and are arguably an improvement on currently accepted methods.</div></div>","PeriodicalId":101075,"journal":{"name":"Research Methods in Applied Linguistics","volume":"4 1","pages":"Article 100197"},"PeriodicalIF":0.0,"publicationDate":"2025-03-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143592104","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Narrative trajectory in identity and subjectivity research in applied linguistics 身份叙事轨迹与应用语言学主体性研究

Research Methods in Applied Linguistics Pub Date : 2025-03-05 DOI: 10.1016/j.rmal.2025.100189

Steven Yeung

{"title":"Narrative trajectory in identity and subjectivity research in applied linguistics","authors":"Steven Yeung","doi":"10.1016/j.rmal.2025.100189","DOIUrl":"10.1016/j.rmal.2025.100189","url":null,"abstract":"<div><div>Studying the trajectory of a social actor has attracted the attention of applied linguists. The construct of trajectory connects one's biographical histories, situated practices and experiences and their ensuing consequences vis-à-vis wider social, cultural, political and economic spaces. Trajectory contributes to identity and subjectivity research by allowing for the examination of evolving identities and subjectivities over time, thereby shedding light on the process of becoming. In identity and subjectivity research, stance is often used as a heuristic. However, stance only captures identity and subjectivity at a particular point in time; it does not help to trace identity trajectories that take place across timescales and events. This paper therefore introduces the idea of <em>narrative trajectory</em> as an analytical approach, drawing on sociolinguistic and linguistic anthropological work on stancetaking, chronotope and interdiscursivity. Social actors take stances through narrated events and narrating events, which can form narrated and narrating trajectories respectively if interdiscursive links between events can be traced. The tracing of such links can be accomplished by attending to various types of signs, models or social positions and relevant spatial-temporal configurations (chronotopes) across multiple narrated and narrating trajectories. Taken together, this multiplicity of trajectories forms an overall narrative trajectory. This paper discusses the tracing of a narrative trajectory with an ethnographic case study of an English learner in Hong Kong. The concept was applied to show how her identity as a struggling learner emerged and evolved across time and space. Recommendations for identity and subjectivity research in applied linguistics are also provided.</div></div>","PeriodicalId":101075,"journal":{"name":"Research Methods in Applied Linguistics","volume":"4 1","pages":"Article 100189"},"PeriodicalIF":0.0,"publicationDate":"2025-03-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143549577","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Modeling rater judgments of interpreting quality: Ordinal logistic regression using neural-based evaluation metrics, acoustic fluency measures, and computational linguistic indices 对口译质量的比较判断建模：使用基于神经的评价指标、声学流畅性测量和计算语言指标的有序逻辑回归

Research Methods in Applied Linguistics Pub Date : 2025-03-05 DOI: 10.1016/j.rmal.2025.100194

Chao Han , Xiaolei Lu , Shirong Chen

{"title":"Modeling rater judgments of interpreting quality: Ordinal logistic regression using neural-based evaluation metrics, acoustic fluency measures, and computational linguistic indices","authors":"Chao Han , Xiaolei Lu , Shirong Chen","doi":"10.1016/j.rmal.2025.100194","DOIUrl":"10.1016/j.rmal.2025.100194","url":null,"abstract":"<div><div>Human raters remain central to interpreting quality assessment (IQA); however, recent years have witnessed a growing body of research exploring automatic assessment. These studies have used machine translation evaluation metrics, acoustic fluency measures, and computational linguistic indices as separate approaches to model rater judgments of interpreting quality. Nonetheless, limited research has integrated these three types of measures within a single framework. To address this gap, this exploratory and proof-of-concept study adopts an integrative approach, combining these three types of measures to model rater judgments of interpreting quality as a classification problem. Using a dataset of 161 Chinese-to-English interpretations, we applied ordinal logistic regression analysis to identify significant predictors across fidelity, fluency, and linguistic dimensions. The analyses yielded two sets of significant predictors, including (a) COMET-22, mean length of unfilled pauses, mean length of run, and mean word length, and (b) BLEURT-20, phonation time ratio, speech rate, mean word length, type-token ratio for content words, type-token ratio for all words, and mean word frequency for content words. These models performed well on the testing dataset, particularly for classifying interpretations into four bands of overall interpreting quality (e.g., accuracy = .643, 1-off accuracy = .805), based on the Rasch-calibrated scores from human evaluation. These findings suggest that this integrated approach may enhance the precision and scalability of IQA and has the potential to reduce logistical burdens in large-scale professional interpreter certification exams and language proficiency tests.</div></div>","PeriodicalId":101075,"journal":{"name":"Research Methods in Applied Linguistics","volume":"4 1","pages":"Article 100194"},"PeriodicalIF":0.0,"publicationDate":"2025-03-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143549576","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Fuzzy-set qualitative comparative analysis (fsQCA) in second language acquisition: An applied example of writing engagement 模糊集定性比较分析（fsQCA）在二语习得中的应用：以写作投入为例

Research Methods in Applied Linguistics Pub Date : 2025-03-04 DOI: 10.1016/j.rmal.2025.100193

Jialing Sun, Yanyan Wang

{"title":"Fuzzy-set qualitative comparative analysis (fsQCA) in second language acquisition: An applied example of writing engagement","authors":"Jialing Sun, Yanyan Wang","doi":"10.1016/j.rmal.2025.100193","DOIUrl":"10.1016/j.rmal.2025.100193","url":null,"abstract":"<div><div>Fuzzy-set qualitative comparative analysis (fsQCA) has great potential in second language acquisition research but remains underutilized, partly due to its complex theory and limited guidance on practical application. This paper fills this gap by providing a comprehensive introduction to this method, demonstrating its value in uncovering nonlinear, asymmetric causal patterns that are often overlooked by traditional methods. Through an empirical example of writing engagement in SLA, the paper walks through the practical steps of using fsQCA, including conditions selection and model construction, data calibration, necessary condition test, configurational analysis, and presentation and explanation of solutions. FsQCA results revealed how distinct configurations of writing emotions, including hope, enjoyment, anxiety and boredom, affect students’ writing engagement. The paper concludes by discussing the prospects of fsQCA in educational research, including its potential integration with other methods, its adaptability to longitudinal research, and its capacity to model mediation and moderation effects. This work provides a valuable guide for researchers seeking to apply fsQCA to explore the complexities of educational phenomena, especially in the context of SLA.</div></div>","PeriodicalId":101075,"journal":{"name":"Research Methods in Applied Linguistics","volume":"4 1","pages":"Article 100193"},"PeriodicalIF":0.0,"publicationDate":"2025-03-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143549575","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

How long is long? Multiperspective qualitative longitudinal research to capture holistic learning experiences 多长是长？多视角定性纵向研究，捕捉整体学习经验

Research Methods in Applied Linguistics Pub Date : 2025-02-28 DOI: 10.1016/j.rmal.2025.100192

Christopher Shepard, Heath Rose

{"title":"How long is long? Multiperspective qualitative longitudinal research to capture holistic learning experiences","authors":"Christopher Shepard, Heath Rose","doi":"10.1016/j.rmal.2025.100192","DOIUrl":"10.1016/j.rmal.2025.100192","url":null,"abstract":"<div><div>Temporality plays a crucial role in education, as learners often engage in formal learning over several years, progressing through various educational stages. However, much of applied linguistics research tends to reduce learning to isolated moments, which are then used to describe an overall developmental process. While longitudinal studies attempt to capture development over time, they often present fragmented snapshots, leaving significant gaps between timepoints. This paper critiques such studies, suggesting that labeling them as ‘longitudinal’ can obscure their failure to fully capture the ongoing temporal nature of learning. Qualitative longitudinal research offers a solution to this limitation by establishing a framework in which the changing effects of time can be better understood. This paper advocates for strengthening this approach by integrating multiperspective research, which involves gathering insights from multiple viewpoints during study design, data collection, and analysis. To illustrate this research approach, data from a study on English-medium instruction (EMI) in Hong Kong is presented. In this study, 26 undergraduate students were tracked throughout their four-year degree programs, supplemented with insights from 20 EMI lecturers and 20 English-language instructors, who offer a broader, more nuanced view of the students' learning experiences and temporal shifts. Together, these perspectives provide insight into temporal changes within broader time frames and reveal students’ overall undergraduate journey as they acclimate (and at times do not acclimate) to the various challenges associated with learning through the medium of English.</div></div>","PeriodicalId":101075,"journal":{"name":"Research Methods in Applied Linguistics","volume":"4 1","pages":"Article 100192"},"PeriodicalIF":0.0,"publicationDate":"2025-02-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143510278","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Putting subjectivity in time: Qualitatively examining multilingual identity development via a Q-inspired longitudinal design 将主观性放在时间上：通过q启发的纵向设计定性地检查多语言身份发展

Research Methods in Applied Linguistics Pub Date : 2025-02-28 DOI: 10.1016/j.rmal.2025.100188

Xinran Wu , Nicola Morea

{"title":"Putting subjectivity in time: Qualitatively examining multilingual identity development via a Q-inspired longitudinal design","authors":"Xinran Wu , Nicola Morea","doi":"10.1016/j.rmal.2025.100188","DOIUrl":"10.1016/j.rmal.2025.100188","url":null,"abstract":"<div><div>The dynamic turn of Applied Linguistics accentuates changes and evolvement, which requires appropriate study designs and methods with a temporal focus to accommodate the complexity in those on-going processes. Inspired by the data collection activity of Q methodology, this paper focuses on the potential of a longitudinal qualitative study design in investigating individual-level multilingual identity (MI) development, which concerns one's relationship with the multilingual world and their multilingual repertoire, and discusses its capacity in capturing and construing changes over time. The proposed design involves multiple rounds of the Q-sort activity and post-sorting interviews conducted at different timepoints with the same participant. We begin with an introduction of the design with attention to its distinction with group level and single-case Q methodology. This is followed by the conceptualisation of MI, its time-contingency and its compatibility with the proposed design. Then, we demonstrate the critical stages regarding its execution and application through a step by-step methodological tutorial. We present and construe the MI developmental trajectory of a L3-learning Chinese high school student based on qualitative analysis of three rounds of Q-sort and interview data collected over one academic year. The example highlights the design's precision and systematicity in capturing the changes of the focal construct and its efficiency in eliciting rich, targeted data. This leads to the discussions about framing of change as multi-levelled and time as relative in MI research. The whole process involves extensive participation and co-construction from the participant, which leads to discussions regarding the ethical considerations.</div></div>","PeriodicalId":101075,"journal":{"name":"Research Methods in Applied Linguistics","volume":"4 1","pages":"Article 100188"},"PeriodicalIF":0.0,"publicationDate":"2025-02-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143510280","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Meaning and function dimensions of linguistic complexity in second language writing 第二语言写作中语言复杂性的意义和功能维度

Research Methods in Applied Linguistics Pub Date : 2025-02-24 DOI: 10.1016/j.rmal.2025.100191

Xiaofei Lu

引用次数: 0