Peer-Grading at Scale with Rank Aggregation

Abstract

Thanks to the wide availability of the internet and personal computing devices, online teaching methods like Massive Open Online Courses (MOOC) are becoming an essential part of modern education. While these methods enable educators to reach much more students, the massive volume of assignments to grade places a heavy burden on the instructors. Most online courses remedy this by restricting the question types to simple forms or performing naive peer-grading. These approaches are either too restricted to capture students' learning level, or require heavy supervision from the instructors to ensure the grades are fair. In this paper, we propose a rank-aggregation-based peer-grading method that estimates the quality of each assignment and the probability that each student is grading unbiasedly. The estimation errors have theoretical upper-bounds, and the bounds can be proved to tighten when the problem size increases, which is confirmed by our numerical experiment.