{"title":"探索使用 ChatGPT 进行修辞步骤分析的潜力:及时完善、少量学习和微调的影响","authors":"","doi":"10.1016/j.jeap.2024.101422","DOIUrl":null,"url":null,"abstract":"<div><p>Rhetorical move-step analysis has wielded considerable influence in the fields of English for Academic/Specific Purposes. To explore the potential of using ChatGPT for automated move-step analysis, this study examines the impact of few-shot learning, prompt refinement, and base model fine-tuning on its accuracy in move-step annotation. Our dataset consisted of the introduction sections of 100 research articles in the field of applied linguistics that have been manually annotated for move-steps based on a modified version of Swales’ (1990) Create-a-Research-Space model, with 80 for training, 10 for validation, and 10 for testing. We formulated an initial prompt that instructed the base model to perform move-step annotation, evaluated it in a zero-shot setting on the validation set, and subsequently refined it with greater specificity. We also fine-tuned the base model on the training set. Evaluation results on the test set showed that few-shot learning and prompt refinement both led to significant albeit relatively small performance improvements, while fine-tuning the base model achieved substantially higher accuracies (92.3% for move and 80.2% for step annotation). Our results highlight the potential of using ChatGPT for discourse-level annotation tasks and have useful implications for EAP pedagogy. They also provide key recommendations for employing ChatGPT in research.</p></div>","PeriodicalId":47717,"journal":{"name":"Journal of English for Academic Purposes","volume":null,"pages":null},"PeriodicalIF":3.1000,"publicationDate":"2024-07-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Exploring the potential of using ChatGPT for rhetorical move-step analysis: The impact of prompt refinement, few-shot learning, and fine-tuning\",\"authors\":\"\",\"doi\":\"10.1016/j.jeap.2024.101422\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<div><p>Rhetorical move-step analysis has wielded considerable influence in the fields of English for Academic/Specific Purposes. To explore the potential of using ChatGPT for automated move-step analysis, this study examines the impact of few-shot learning, prompt refinement, and base model fine-tuning on its accuracy in move-step annotation. Our dataset consisted of the introduction sections of 100 research articles in the field of applied linguistics that have been manually annotated for move-steps based on a modified version of Swales’ (1990) Create-a-Research-Space model, with 80 for training, 10 for validation, and 10 for testing. We formulated an initial prompt that instructed the base model to perform move-step annotation, evaluated it in a zero-shot setting on the validation set, and subsequently refined it with greater specificity. We also fine-tuned the base model on the training set. Evaluation results on the test set showed that few-shot learning and prompt refinement both led to significant albeit relatively small performance improvements, while fine-tuning the base model achieved substantially higher accuracies (92.3% for move and 80.2% for step annotation). Our results highlight the potential of using ChatGPT for discourse-level annotation tasks and have useful implications for EAP pedagogy. They also provide key recommendations for employing ChatGPT in research.</p></div>\",\"PeriodicalId\":47717,\"journal\":{\"name\":\"Journal of English for Academic Purposes\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":3.1000,\"publicationDate\":\"2024-07-17\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Journal of English for Academic Purposes\",\"FirstCategoryId\":\"98\",\"ListUrlMain\":\"https://www.sciencedirect.com/science/article/pii/S1475158524000900\",\"RegionNum\":1,\"RegionCategory\":\"文学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q1\",\"JCRName\":\"EDUCATION & EDUCATIONAL RESEARCH\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of English for Academic Purposes","FirstCategoryId":"98","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S1475158524000900","RegionNum":1,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"EDUCATION & EDUCATIONAL RESEARCH","Score":null,"Total":0}
Exploring the potential of using ChatGPT for rhetorical move-step analysis: The impact of prompt refinement, few-shot learning, and fine-tuning
Rhetorical move-step analysis has wielded considerable influence in the fields of English for Academic/Specific Purposes. To explore the potential of using ChatGPT for automated move-step analysis, this study examines the impact of few-shot learning, prompt refinement, and base model fine-tuning on its accuracy in move-step annotation. Our dataset consisted of the introduction sections of 100 research articles in the field of applied linguistics that have been manually annotated for move-steps based on a modified version of Swales’ (1990) Create-a-Research-Space model, with 80 for training, 10 for validation, and 10 for testing. We formulated an initial prompt that instructed the base model to perform move-step annotation, evaluated it in a zero-shot setting on the validation set, and subsequently refined it with greater specificity. We also fine-tuned the base model on the training set. Evaluation results on the test set showed that few-shot learning and prompt refinement both led to significant albeit relatively small performance improvements, while fine-tuning the base model achieved substantially higher accuracies (92.3% for move and 80.2% for step annotation). Our results highlight the potential of using ChatGPT for discourse-level annotation tasks and have useful implications for EAP pedagogy. They also provide key recommendations for employing ChatGPT in research.
期刊介绍:
The Journal of English for Academic Purposes provides a forum for the dissemination of information and views which enables practitioners of and researchers in EAP to keep current with developments in their field and to contribute to its continued updating. JEAP publishes articles, book reviews, conference reports, and academic exchanges in the linguistic, sociolinguistic and psycholinguistic description of English as it occurs in the contexts of academic study and scholarly exchange itself.