Using AI-supported peer review to enhance feedback literacy: An investigation of students' revision of feedback on peers' essays

As a vital learning activity in second language (L2) writing classrooms, peer feedback plays a crucial role in improving students' writing skills. However, student reviewers face challenges in providing impactful feedback on peers' essays. Low-quality peer reviews emerge as a persistent problem, adversely affecting the learning effect of peer feedback. To enhance students' peer feedback provision, this study introduces EvaluMate, an AI-supported peer review system, which incorporates a chatbot named Eva, designed to evaluate and provide feedback on student reviewers' comments on peers' essays. Forty-four Chinese undergraduate students engaged with EvaluMate, utilising its features to generate feedback on peers' English argumentative essays. Chat log data capturing the students' interactions with the chatbot were collected, including the comments they wrote on peer essays and the feedback offered by the chatbot on their comments. The results indicate that the integration of AI supervision improved the quality of students' peer reviews. Students employed various strategies during their comment revision in response to AI feedback, such as introducing new points, adding details, and providing illustrative examples, which helped improve their comment quality. These findings shed light on the benefits of AI-supported peer review systems in empowering students to provide more valuable feedback on peers' written work.

Practitioner notes

What is already known about this topic

• Scholars have extensively investigated diverse pedagogical strategies to enhance students' peer feedback provision skills in second language (L2) writing classrooms.
• Artificial intelligence (AI) technologies have been utilised to monitor and evaluate the peer feedback generated by student reviewers.
• AI-enabled peer feedback evaluation tools have demonstrated the ability to provide valid assessments of student reviewers' peer feedback.

What this paper adds

• In the context of L2 writing, there is a lack of bespoke AI-enabled peer feedback evaluation tools. To address this gap, we have developed an AI-supported peer review system, EvaluMate, which incorporates a large language model-based chatbot named Eva. Eva is designed to provide feedback on L2 students' comments on their peers' writing.
• While previous studies have primarily focused on assessing the validity of AI-enabled peer feedback evaluation tools, little is known about how students incorporate AI support into improving their peer review comments. To bridge this gap, our study examines not only whether using the system (EvaluMate) can enhance the quality of L2 students' peer review comments but also how students respond to Eva's feedback when revising their comments.

Implications for practice and/or policy

• The development of the AI-supported peer review system (EvaluMate) introduces an innovative pedagogical approach for L2 writing teachers to train and enhance their students' peer feedback provision skills.
• Integrating AI supervision into L2 students' peer feedback generation improves the quality of comments provided by student reviewers on their peers' writing.
• Students employ various strategies when revising their comments in response to Eva's feedback, and these strategies result in varying degrees of improvement in comment quality. L2 writing teachers can teach effective revision strategies to their students.