Towards LLM-assisted move annotation: Leveraging ChatGPT-4 to analyse the genre structure of CEO statements in corporate social responsibility reports

Abstract

Recent studies have explored the potential of large language models (LLMs) in annotating rhetorical moves in academic genres such as research article abstracts and research article introductions. Extending this line of research beyond academic contexts, this study investigates the feasibility of using ChatGPT-4 to automate move annotation in a professional genre: CEO statements from corporate social responsibility reports. The study proceeded in two phases. First, 30 CEO statements were used for move identification, prompt design and validation, resulting in a prompt optimized for the move annotation task. Subsequently, a corpus of 50 CEO statements was used to further test the efficacy of the model conditioned by the established prompt. The assessment showed that the model's annotation outputs yielded an accuracy of 87.14 %. However, the double annotation rounds revealed an intra-model inconsistency rate of 32.87 %, indicating the need for human verification in inconsistent cases. This study demonstrates that LLMs can effectively support the development of move-annotated corpora through reduced workload and enhanced methodological duplicability. These annotated corpora serve as valuable pedagogical materials for ESP practitioners in genre-based instruction. Furthermore, the integration of LLMs can aid in improving researchers' own thinking, thereby contributing to potential theory refinement in the field.