Research Methods in Applied Linguistics最新文献_第3页

Modeling rater judgments of interpreting quality: Ordinal logistic regression using neural-based evaluation metrics, acoustic fluency measures, and computational linguistic indices 对口译质量的比较判断建模：使用基于神经的评价指标、声学流畅性测量和计算语言指标的有序逻辑回归

Research Methods in Applied Linguistics Pub Date : 2025-03-05 DOI: 10.1016/j.rmal.2025.100194

Chao Han , Xiaolei Lu , Shirong Chen

{"title":"Modeling rater judgments of interpreting quality: Ordinal logistic regression using neural-based evaluation metrics, acoustic fluency measures, and computational linguistic indices","authors":"Chao Han , Xiaolei Lu , Shirong Chen","doi":"10.1016/j.rmal.2025.100194","DOIUrl":"10.1016/j.rmal.2025.100194","url":null,"abstract":"<div><div>Human raters remain central to interpreting quality assessment (IQA); however, recent years have witnessed a growing body of research exploring automatic assessment. These studies have used machine translation evaluation metrics, acoustic fluency measures, and computational linguistic indices as separate approaches to model rater judgments of interpreting quality. Nonetheless, limited research has integrated these three types of measures within a single framework. To address this gap, this exploratory and proof-of-concept study adopts an integrative approach, combining these three types of measures to model rater judgments of interpreting quality as a classification problem. Using a dataset of 161 Chinese-to-English interpretations, we applied ordinal logistic regression analysis to identify significant predictors across fidelity, fluency, and linguistic dimensions. The analyses yielded two sets of significant predictors, including (a) COMET-22, mean length of unfilled pauses, mean length of run, and mean word length, and (b) BLEURT-20, phonation time ratio, speech rate, mean word length, type-token ratio for content words, type-token ratio for all words, and mean word frequency for content words. These models performed well on the testing dataset, particularly for classifying interpretations into four bands of overall interpreting quality (e.g., accuracy = .643, 1-off accuracy = .805), based on the Rasch-calibrated scores from human evaluation. These findings suggest that this integrated approach may enhance the precision and scalability of IQA and has the potential to reduce logistical burdens in large-scale professional interpreter certification exams and language proficiency tests.</div></div>","PeriodicalId":101075,"journal":{"name":"Research Methods in Applied Linguistics","volume":"4 1","pages":"Article 100194"},"PeriodicalIF":0.0,"publicationDate":"2025-03-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143549576","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Fuzzy-set qualitative comparative analysis (fsQCA) in second language acquisition: An applied example of writing engagement 模糊集定性比较分析（fsQCA）在二语习得中的应用：以写作投入为例

Research Methods in Applied Linguistics Pub Date : 2025-03-04 DOI: 10.1016/j.rmal.2025.100193

Jialing Sun, Yanyan Wang

{"title":"Fuzzy-set qualitative comparative analysis (fsQCA) in second language acquisition: An applied example of writing engagement","authors":"Jialing Sun, Yanyan Wang","doi":"10.1016/j.rmal.2025.100193","DOIUrl":"10.1016/j.rmal.2025.100193","url":null,"abstract":"<div><div>Fuzzy-set qualitative comparative analysis (fsQCA) has great potential in second language acquisition research but remains underutilized, partly due to its complex theory and limited guidance on practical application. This paper fills this gap by providing a comprehensive introduction to this method, demonstrating its value in uncovering nonlinear, asymmetric causal patterns that are often overlooked by traditional methods. Through an empirical example of writing engagement in SLA, the paper walks through the practical steps of using fsQCA, including conditions selection and model construction, data calibration, necessary condition test, configurational analysis, and presentation and explanation of solutions. FsQCA results revealed how distinct configurations of writing emotions, including hope, enjoyment, anxiety and boredom, affect students’ writing engagement. The paper concludes by discussing the prospects of fsQCA in educational research, including its potential integration with other methods, its adaptability to longitudinal research, and its capacity to model mediation and moderation effects. This work provides a valuable guide for researchers seeking to apply fsQCA to explore the complexities of educational phenomena, especially in the context of SLA.</div></div>","PeriodicalId":101075,"journal":{"name":"Research Methods in Applied Linguistics","volume":"4 1","pages":"Article 100193"},"PeriodicalIF":0.0,"publicationDate":"2025-03-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143549575","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

How long is long? Multiperspective qualitative longitudinal research to capture holistic learning experiences 多长是长？多视角定性纵向研究，捕捉整体学习经验

Research Methods in Applied Linguistics Pub Date : 2025-02-28 DOI: 10.1016/j.rmal.2025.100192

Christopher Shepard, Heath Rose

{"title":"How long is long? Multiperspective qualitative longitudinal research to capture holistic learning experiences","authors":"Christopher Shepard, Heath Rose","doi":"10.1016/j.rmal.2025.100192","DOIUrl":"10.1016/j.rmal.2025.100192","url":null,"abstract":"<div><div>Temporality plays a crucial role in education, as learners often engage in formal learning over several years, progressing through various educational stages. However, much of applied linguistics research tends to reduce learning to isolated moments, which are then used to describe an overall developmental process. While longitudinal studies attempt to capture development over time, they often present fragmented snapshots, leaving significant gaps between timepoints. This paper critiques such studies, suggesting that labeling them as ‘longitudinal’ can obscure their failure to fully capture the ongoing temporal nature of learning. Qualitative longitudinal research offers a solution to this limitation by establishing a framework in which the changing effects of time can be better understood. This paper advocates for strengthening this approach by integrating multiperspective research, which involves gathering insights from multiple viewpoints during study design, data collection, and analysis. To illustrate this research approach, data from a study on English-medium instruction (EMI) in Hong Kong is presented. In this study, 26 undergraduate students were tracked throughout their four-year degree programs, supplemented with insights from 20 EMI lecturers and 20 English-language instructors, who offer a broader, more nuanced view of the students' learning experiences and temporal shifts. Together, these perspectives provide insight into temporal changes within broader time frames and reveal students’ overall undergraduate journey as they acclimate (and at times do not acclimate) to the various challenges associated with learning through the medium of English.</div></div>","PeriodicalId":101075,"journal":{"name":"Research Methods in Applied Linguistics","volume":"4 1","pages":"Article 100192"},"PeriodicalIF":0.0,"publicationDate":"2025-02-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143510278","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Putting subjectivity in time: Qualitatively examining multilingual identity development via a Q-inspired longitudinal design 将主观性放在时间上：通过q启发的纵向设计定性地检查多语言身份发展

Research Methods in Applied Linguistics Pub Date : 2025-02-28 DOI: 10.1016/j.rmal.2025.100188

Xinran Wu , Nicola Morea

{"title":"Putting subjectivity in time: Qualitatively examining multilingual identity development via a Q-inspired longitudinal design","authors":"Xinran Wu , Nicola Morea","doi":"10.1016/j.rmal.2025.100188","DOIUrl":"10.1016/j.rmal.2025.100188","url":null,"abstract":"<div><div>The dynamic turn of Applied Linguistics accentuates changes and evolvement, which requires appropriate study designs and methods with a temporal focus to accommodate the complexity in those on-going processes. Inspired by the data collection activity of Q methodology, this paper focuses on the potential of a longitudinal qualitative study design in investigating individual-level multilingual identity (MI) development, which concerns one's relationship with the multilingual world and their multilingual repertoire, and discusses its capacity in capturing and construing changes over time. The proposed design involves multiple rounds of the Q-sort activity and post-sorting interviews conducted at different timepoints with the same participant. We begin with an introduction of the design with attention to its distinction with group level and single-case Q methodology. This is followed by the conceptualisation of MI, its time-contingency and its compatibility with the proposed design. Then, we demonstrate the critical stages regarding its execution and application through a step by-step methodological tutorial. We present and construe the MI developmental trajectory of a L3-learning Chinese high school student based on qualitative analysis of three rounds of Q-sort and interview data collected over one academic year. The example highlights the design's precision and systematicity in capturing the changes of the focal construct and its efficiency in eliciting rich, targeted data. This leads to the discussions about framing of change as multi-levelled and time as relative in MI research. The whole process involves extensive participation and co-construction from the participant, which leads to discussions regarding the ethical considerations.</div></div>","PeriodicalId":101075,"journal":{"name":"Research Methods in Applied Linguistics","volume":"4 1","pages":"Article 100188"},"PeriodicalIF":0.0,"publicationDate":"2025-02-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143510280","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Meaning and function dimensions of linguistic complexity in second language writing 第二语言写作中语言复杂性的意义和功能维度

Research Methods in Applied Linguistics Pub Date : 2025-02-24 DOI: 10.1016/j.rmal.2025.100191

Xiaofei Lu

引用次数: 0

Developing a cross-case, time-ordered analysis of informal language learning from ethnographic narratives 从民族志叙事中发展非正式语言学习的跨案例、时间顺序分析

Research Methods in Applied Linguistics Pub Date : 2025-02-22 DOI: 10.1016/j.rmal.2025.100190

Mark Dressman , Denyze Toffoli , Ju Seong Lee

引用次数: 0

Engaging young language learners in participatory research: Visual arts-based approach 让年轻的语言学习者参与研究：基于视觉艺术的方法

Research Methods in Applied Linguistics Pub Date : 2025-02-18 DOI: 10.1016/j.rmal.2025.100187

Junjie Li, Weizhao Gong

{"title":"Engaging young language learners in participatory research: Visual arts-based approach","authors":"Junjie Li, Weizhao Gong","doi":"10.1016/j.rmal.2025.100187","DOIUrl":"10.1016/j.rmal.2025.100187","url":null,"abstract":"<div><div>This paper explores the visual arts-based approach to participatory research with children, aged 9 to 11, by analysing the micro-processes of children agentively making meanings in individual and group interviews. Specifically, drawing and graphing are utilised to facilitate participants’ thinking, verbalisation, and nonverbal expressions during data elicitation. We draw on two example qualitative studies to discuss the innovative methods. By interpreting critical episodes emerging from visual arts-crafting activities, we find a mixed set of visual tools empowering in listening to children's voices in hierarchical bilingual educational contexts, including two schools and a private language institution.</div><div>The methodological insights are examined in line with two overarching themes: a) the mediational and scaffolding role of visual artefacts in children's embodied meaning-making processes, and b) children's strategic appropriation of activities to address their own personal and social goals. Findings indicate that, as local actors, young participants agentively claim expertise resorting to ‘insider knowledge’ of childhood, subvert the prescribed power status in relation to adult researchers, and strategically appropriate research instruments for reinterpreting the research agenda. We argue a glimpse of children's multi-faceted lifeworlds and peer culture is temporarily made possible by means of visual arts-based approach, generating data that is otherwise hard to elicit or interpret.</div></div>","PeriodicalId":101075,"journal":{"name":"Research Methods in Applied Linguistics","volume":"4 1","pages":"Article 100187"},"PeriodicalIF":0.0,"publicationDate":"2025-02-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143429106","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Mobile-assisted language learning with commercial apps: A focused methodological review of quantitative/mixed methods research and ethics 商业应用的移动辅助语言学习：定量/混合方法研究和伦理的重点方法论回顾

Research Methods in Applied Linguistics Pub Date : 2025-01-31 DOI: 10.1016/j.rmal.2025.100186

Matt Kessler, Tania Ferronato, María José Torres Centurion, Melike Akay, Jihye Kim

{"title":"Mobile-assisted language learning with commercial apps: A focused methodological review of quantitative/mixed methods research and ethics","authors":"Matt Kessler, Tania Ferronato, María José Torres Centurion, Melike Akay, Jihye Kim","doi":"10.1016/j.rmal.2025.100186","DOIUrl":"10.1016/j.rmal.2025.100186","url":null,"abstract":"<div><div>Researchers have increasingly investigated the effectiveness of commercial mobile-assisted language learning (MALL) applications (apps) such as Duolingo, Babbel, Busuu, Memrise, and Rosetta Stone, with several methodological reviews that have been published to date. However, prior reviews have often lumped together MALL studies involving commercial apps with those that investigate teacher-generated MALL activities. Such reviews also provide limited discussions of important issues involving research ethics. This study aims to fill these gaps by conducting a focused review of studies from the past 10 years (2014–2023) that have appeared in six key CALL journals. Investigated are 1) the research methods used in commercial MALL app studies using quantitative or mixed methods (e.g., samples, target languages, instruments), and 2) the extent to which researchers have discussed issues of research ethics (e.g., disclosing funding, conflicts of interest). The findings illustrate trends in the popularity of certain target languages, topics, and instruments, along with issues involving the reporting of key statistical information. Some authors also provide wide-ranging discussions of ethics; however, many studies lack transparency concerning potential conflicts of interest. Based on these findings, recommendations are provided for conducting future research with commercial MALL apps.</div></div>","PeriodicalId":101075,"journal":{"name":"Research Methods in Applied Linguistics","volume":"4 1","pages":"Article 100186"},"PeriodicalIF":0.0,"publicationDate":"2025-01-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143101669","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Beyond BLEU: Repurposing neural-based metrics to assess interlingual interpreting in tertiary-level language learning settings 超越BLEU：重新利用基于神经的指标来评估三级语言学习环境中的语际口译

Research Methods in Applied Linguistics Pub Date : 2025-01-16 DOI: 10.1016/j.rmal.2025.100184

Chao Han , Xiaolei Lu

{"title":"Beyond BLEU: Repurposing neural-based metrics to assess interlingual interpreting in tertiary-level language learning settings","authors":"Chao Han , Xiaolei Lu","doi":"10.1016/j.rmal.2025.100184","DOIUrl":"10.1016/j.rmal.2025.100184","url":null,"abstract":"<div><div>Recent years have seen a revival of using translation and interpreting (T&I) as a pedagogical and assessment tool to enhance language learning. This growing usage contributes to an increasing amount of learner-generated T&I data, creating a strong demand for assessment. To alleviate this issue, researchers have proposed repurposing machine translation (MT) evaluation metrics to automatically assess human-generated T&I. In this article, we report on the first large-scale study in which we leveraged sophisticated neural-based MT evaluation metrics for automatically assessing English-Chinese interpreting, using a database called <em>Interpreting Quality Evaluation Corpus</em>. To evaluate the efficacy of neural-based metrics, we correlated them with human benchmark scores. Because of the unique data structure, we conducted an internal meta-analysis of correlation coefficients to examine the overall machine-human correlation, and further performed meta-regression to identify potential significant moderators. We find that: a) the overall meta-synthesized correlations were fairly strong: <em>r</em> = .652 and <em>r<sub>s</sub></em> = .631; b) the type of neural-based metrics was a significant moderator, with BLEURT-20 registering the highest correlations (<em>r</em> = .738, <em>r<sub>s</sub></em> = .700); and c) the level of human rater reliability was also a significant moderator. We discussed these findings and their implications for T&I assessment in higher education.</div></div>","PeriodicalId":101075,"journal":{"name":"Research Methods in Applied Linguistics","volume":"4 1","pages":"Article 100184"},"PeriodicalIF":0.0,"publicationDate":"2025-01-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143101668","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Towards more appropriate modelling of linguistic complexity measures: Beyond traditional regression models 走向更合适的语言复杂性度量建模：超越传统的回归模型

Research Methods in Applied Linguistics Pub Date : 2025-01-16 DOI: 10.1016/j.rmal.2025.100182

Akira Murakami

{"title":"Towards more appropriate modelling of linguistic complexity measures: Beyond traditional regression models","authors":"Akira Murakami","doi":"10.1016/j.rmal.2025.100182","DOIUrl":"10.1016/j.rmal.2025.100182","url":null,"abstract":"<div><div>Despite a recent emphasis on appropriate quantitative data analysis as part of the methodological reform, applied linguists often overlook scrutinising the adequacy of their analytical methods, risking model misspecification. This article critiques the use of regression models assuming normal error distributions for modelling linguistic complexity measures. It examines two alternative approaches: weighted linear regression with a log-transformed outcome variable and negative binomial regression, demonstrating how they mitigate associated limitations. Normal error regression models are inadequate for modelling count-based ratio variables due to two main issues: (i) count variables are theoretically lower-bounded, a constraint not addressed by normal error regression models, and (ii) variations in count quantities lead to differences in sampling variability, violating the homoscedasticity assumption and potentially inflating the Type I error rate. Analysis of 14 syntactic complexity measures and an artificial-data simulation show that the lower bound of the prediction intervals for normal error regression models often falls below the theoretical minimum in realistic scenarios. Moreover, the denominator count of syntactic complexity measures negatively correlates with variability, causing heteroscedasticity, a higher false-positive rate, and reduced true value coverage by 80% confidence intervals. Both alternative approaches outperform normal error regression models in these criteria. These findings challenge the suitability of normal error regressions for modelling syntactic complexity measures and caution against their use for other count-based measures in second language research and corpus linguistics. This necessitates reevaluating the widespread use of normal error regression in applied linguistics, urging methodologists to develop and validate more structurally faithful modelling approaches.</div></div>","PeriodicalId":101075,"journal":{"name":"Research Methods in Applied Linguistics","volume":"4 1","pages":"Article 100182"},"PeriodicalIF":0.0,"publicationDate":"2025-01-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143101667","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0