Aleph: An indicator of decision-making patterns within collaborative marking sessions for writing assessment that can predict reliability

Abstract

Assessing writing is difficult due to language inherent subjectivity. Rubrics can help tackle this problem, although collaborative training sessions on their interpretation are required in order to reach consensus between markers and ensure minimum reliability levels. This article reports the construction of Aleph; an indicator built on raters’ decision-making behavior in collaborative environments, which can explain agreement between markers. Evaluated within real training sessions of three markers for a high-stake written test, it showed excellent results on identifying markers’ patterns and a good correlation with Fleiss’ Kappa on agreement.