PeerBERT: Automated Characterization of Peer Review Comments across Courses

Abstract

Writing-to-learn pedagogies are an evidence-based practice known to aid students in constructing knowledge. Barriers exist for the implementation of such assignments; namely, instructors feel they do not have time to provide each student with feedback. To ease implementation of writing-to-learn assignments at scale, we have incorporated automated peer review, which facilitates peer review without input from the instructor. Participating in peer review can positively impact students’ learning and allow students to receive feedback on their writing. Instructors may want to monitor these peer interactions and gain insight into their students’ understanding using the feedback generated by their peers. To facilitate instructors’ use of the content from students’ peer review comments, we pre-trained a transformer model called PeerBERT. PeerBERT was fine-tuned on several downstream tasks to categorize students’ peer review comments as praise, problem/solution, or verification/summary. The model exhibits high accuracy, even across different peer review prompts, assignments, and courses. Additional downstream tasks label problem/solution peer review comments as one or more types: writing/formatting, missing content/needs elaboration, and incorrect content. This approach can help instructors pinpoint common issues in student writing by parsing out which comments are problem/solution and which type of problem/solution students identify.