{"title":"Exploring the potential of using ChatGPT for rhetorical move-step analysis: The impact of prompt refinement, few-shot learning, and fine-tuning","authors":"Minjin Kim, Xiaofei Lu","doi":"10.1016/j.jeap.2024.101422","DOIUrl":null,"url":null,"abstract":"<div><p>Rhetorical move-step analysis has wielded considerable influence in the fields of English for Academic/Specific Purposes. To explore the potential of using ChatGPT for automated move-step analysis, this study examines the impact of few-shot learning, prompt refinement, and base model fine-tuning on its accuracy in move-step annotation. Our dataset consisted of the introduction sections of 100 research articles in the field of applied linguistics that have been manually annotated for move-steps based on a modified version of Swales’ (1990) Create-a-Research-Space model, with 80 for training, 10 for validation, and 10 for testing. We formulated an initial prompt that instructed the base model to perform move-step annotation, evaluated it in a zero-shot setting on the validation set, and subsequently refined it with greater specificity. We also fine-tuned the base model on the training set. Evaluation results on the test set showed that few-shot learning and prompt refinement both led to significant albeit relatively small performance improvements, while fine-tuning the base model achieved substantially higher accuracies (92.3% for move and 80.2% for step annotation). Our results highlight the potential of using ChatGPT for discourse-level annotation tasks and have useful implications for EAP pedagogy. They also provide key recommendations for employing ChatGPT in research.</p></div>","PeriodicalId":47717,"journal":{"name":"Journal of English for Academic Purposes","volume":"71 ","pages":"Article 101422"},"PeriodicalIF":3.1000,"publicationDate":"2024-07-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of English for Academic Purposes","FirstCategoryId":"98","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S1475158524000900","RegionNum":1,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"EDUCATION & EDUCATIONAL RESEARCH","Score":null,"Total":0}
引用次数: 0
Abstract
Rhetorical move-step analysis has wielded considerable influence in the fields of English for Academic/Specific Purposes. To explore the potential of using ChatGPT for automated move-step analysis, this study examines the impact of few-shot learning, prompt refinement, and base model fine-tuning on its accuracy in move-step annotation. Our dataset consisted of the introduction sections of 100 research articles in the field of applied linguistics that have been manually annotated for move-steps based on a modified version of Swales’ (1990) Create-a-Research-Space model, with 80 for training, 10 for validation, and 10 for testing. We formulated an initial prompt that instructed the base model to perform move-step annotation, evaluated it in a zero-shot setting on the validation set, and subsequently refined it with greater specificity. We also fine-tuned the base model on the training set. Evaluation results on the test set showed that few-shot learning and prompt refinement both led to significant albeit relatively small performance improvements, while fine-tuning the base model achieved substantially higher accuracies (92.3% for move and 80.2% for step annotation). Our results highlight the potential of using ChatGPT for discourse-level annotation tasks and have useful implications for EAP pedagogy. They also provide key recommendations for employing ChatGPT in research.
期刊介绍:
The Journal of English for Academic Purposes provides a forum for the dissemination of information and views which enables practitioners of and researchers in EAP to keep current with developments in their field and to contribute to its continued updating. JEAP publishes articles, book reviews, conference reports, and academic exchanges in the linguistic, sociolinguistic and psycholinguistic description of English as it occurs in the contexts of academic study and scholarly exchange itself.