Evaluating the quality of student-generated content in learnersourcing: A large language model based approach

Abstract

In learnersoucing, automatic evaluation of student-generated content (SGC) is significant as it streamlines the evaluation process, provides timely feedback, and enhances the objectivity of grading, ultimately supporting more effective and efficient learning outcomes. However, the methods of aggregating students’ evaluations of SGC face the problems of inefficiency and cold start. The methods of combining feature engineering and deep learning suffer from the problems of insufficient accuracy and low scalability. This study introduced an automated SGC quality evaluation method based on a large language model (LLM). The method made a comprehensive evaluation by allowing LLM to simulate the cognitive process of human evaluation through the Reason-Act-Evaluate (RAE) prompt and integrating an assisted model to analyze the external features of SGCs. The study utilized the SGCs in a learnersourcing platform to experiment with the feasibility of the method. The results showed that LLM is able to achieve high agreement with experts on the quality evaluation of SGC through RAE prompt, and better results can be achieved with the help of assisted models.