Methodological rigor in quantitative L2 research: A focus on interventionist experimental studies

Abstract

An increasing bulk of research highlights the importance of robust methodologies in ensuring the validity and reliability of research findings; yet, concerns remain regarding the quality of quantitative studies within second language (L2) research. To address the gap, this study aimed to systematically analyze a corpus of quantitative articles to assess the extent to which they adhere to established methodological best practices. The corpus comprises 791 interventionist quantitative articles published over 12 years in 8 journals selected based on their high impact factor and relevance to key areas within applied linguistics. A detailed coding scheme was developed to evaluate the articles across several crucial methodological dimensions, including sampling and design issues, types of statistical analyses, the necessary statistical assumptions to be checked, reporting practices, and visual presentation of data. The findings revealed that while improvements were evident in some areas, such as design-related issues and reporting practices, there is still a lack/shortage of attention to sampling issues like power analysis, practicing data sharing, and using data-rich/accountable visuals, with no significant improvement over time. These results highlighted areas where improvements in methodological rigor are needed to enhance the credibility and generalizability of quantitative research in applied linguistics. The study promotes best practices in research design and reporting, informs the development of guidelines for future research, and fosters a more critical and reflective approach to interpreting quantitative findings in the field.