Optimizing automated scoring in ILSAs with prompt compression

Abstract

Automated scoring (AS) has become increasingly prevalent in educational measurement. However, applying it to international reading assessments remains challenging, particularly due to the length and complexity of the required prompting, driven by the need to include lengthy reading passages and detailed scoring guides. Processing these lengthy inputs results in high computational costs and may impede the performance of large language models (LLMs). This study explored the potential of optimizing AS with prompt compression using OpenAI's LLM, GPT-4o. Our results show that prompt compression significantly reduces the length of reading passages and scoring guides while maintaining their essential content. Reading passages and scoring guides were compressed to approximately 18% and 15% of their original lengths, respectively. Despite this substantial compression, the AS showed remarkable performance, with an accuracy of 92.87% and a kappa score of 0.8041, closely approximating the results obtained without compression. These findings suggest optimizing AS with prompt compression can improve its efficiency and scalability, particularly in international reading assessments.