Genre-based fine-tuning of large language models with self-organizing maps for automated writing evaluation

Abstract

Automated Writing Evaluation (AWE) systems have significantly advanced in providing feedback for academic essay writing. However, their predominant focus on sentence-level features highlights the need for a broader approach to AWE development. While genre-based AWE systems aim to address the socio-rhetorical complexities of writing for specific audiences and purposes, their availability remains limited. This scarcity is largely due to methodological constraints in developing robust feedback engines that effectively support discipline-specific writing needs. This article describes a new method for fine-tuning large-language models (LLM) and evaluating model performance, which we refer to as G-FiT Mapping (Genre-based FIne-Tuning with self-organizing maps). This method utilizes semi-automated annotation of genre-based functional-rhetorical units of text to efficiently fine-tune an LLM and then uses self-organizing maps to evaluate and improve network performance. The G-FiT Mapping method resulted in a new automated feedback engine for an intelligent tutoring system called Dissemity, for DISSeminating research with clarITY, that supports discipline-specific, scientific writers in writing for publication. We demonstrate use of G-Fit Mapping for establishing measurable improvements in network performance, offering implications for network interpretation, genre-based AWE, and AI-based learning systems development.