REFERENT: Transformer-based Feedback Generation using Assignment Information for Programming Course

Abstract

Students require feedback on programming assignments to improve their programming skills. An Automated feedback generation (AFG) technique proposes to provide feedback-corrected submissions for incorrect student programming submissions in programming courses. However, these techniques are limited as they rely on the availability of correct submissions as a reference to generate feedback. In situations where correct submissions are not available, they resort to using mutation operators, which can lead to a search space explosion problem. In this work, we propose REFERENT, Transformer-based feedback generation using assignment information. REFERENT uses transfer learning on a pre-trained model with data from students’ submission history from the past assignment. To generate assignment-related feedback, we use a title, tag, assignment description, and test case as assignment information. REFERENT can generate feedback without a reference program in limited resources. We conducted a preliminary study to confirm the effectiveness of REFERENT and the feasibility of using assignment information. REFERENT generated feedback for 32.7% of incorrect submissions without reference programs and that its performance increased up to 50.7% when reference programs were used. We also check whether the submission history, assignment information, and repair knowledge of open-source software help generate feedback.