A High-Quality Generation Approach for Educational Programming Projects Using LLM

Abstract

High-quality programming projects for education are critically required in teaching. However, it is hard to develop those projects efficiently and artificially constrained by the lecturers' experience and background. The recent popularity of large language models (LLMs) has led to a great number of applications in the field of education, but concerns persist that the output might be unreliable when dealing with intricate requirements. In this study, we design a customized role-based agent (CRBA), which can be configured for different roles specializing in specific areas of expertise, making the LLM yield content of higher specialization. An iterative architecture of multi-CRBAs is proposed to generate multistep projects, where CRBAs automatically criticize and optimize the LLM's intermediate outputs to enhance quality. We propose ten evaluation metrics across three aspects to assess project quality through expert grading. Further, we conduct an A/B test among 60 undergraduate students in a programming course and collect their feedback through a questionnaire. According to the students' rating results, the LLM-generated projects have comparable performance to man-made ones in terms of project description, learning step setting, assistance to students, and overall project quality. This study effectively integrates LLM into educational scenarios and enhances the efficiency of creating high-quality and practical programming exercises for lecturers.