A Method to Predict Interrater Agreement in Writing Assessment Based on Raters' Individual Differences

Abstract

Writing has been increasingly assessed within educational scenarios. However, assessing writing is a highly subjective task, making the grading of such tests a costly process that demands several graders for calculating interrater agreement to ensure reliability. A drawback of this approach is that interrater agreement does not explain the source of markers' differences; therefore, when agreement is low, the marking process must be repeated, and markers retrained. This article proposes a method to predict agreement, based on markers' individual differences using item response theory. Results showed that the method accurately predicts agreement even with partial evidence.