Enhancing language models with boosting and targeted fine-tuning for real-word error detection

Abstract

Over the past years, extensive research has led to significant advancements in tools for the automatic detection and correction of errors in documents. Despite this progress, several challenges remain unresolved. In particular, the identification of real-word errors – errors involving words that are grammatically valid but contextually inappropriate within a given sentence – continues to pose a considerable difficulty. Addressing such errors requires models with a sophisticated understanding of linguistic context. Transformer-based language models are particularly well-suited for this task due to their contextual modeling capabilities. To further enhance their performance, we propose a boosting-based training approach in conjunction with a synthetically generated data set created via pattern-based noise injection. We evaluate this method across three transformer-based architectures, viz. mBERT, LLaMA 3, and Mistral 7B. Our experimental results show that the boosting-based strategy consistently improves real-word error detection across all models. A subsequent in-depth error analysis reveals limitations in the synthetic training data, prompting the development of a targeted fine-tuning procedure designed to address these shortcomings and further optimize model performance. A comparison with prompt-based inference using a large language model demonstrates that specialized, fine-tuned models yield more reliable performance for this task. Finally, an evaluation under realistic class imbalance highlights practical trade-offs between ranking quality and threshold-based detection, particularly for rare error types.