Cost and Benefit of a Computer Supported Real Time Delphi Method on Inter-Rater Reliability Using Rubrics in Highly Subjective Evaluation Tasks

Abstract

In this paper we evaluate the impact that a Computer Supported Collaborative Marking (CSCM) system has on the marking process of a highly subjective judging task. To do so, we evaluated the inter-rater reliability of reviewers in charge of marking written essays and the time required to perform such task. We analyzed the results of two groups of markers: the experimental group first marked 5 essays in a collaborative way (using the CSCM system, aiming to reach consensus) and the control group marked the same set of essays in a cooperative manner (e.g. separately). Next, both groups corrected the remaining 20 essays in a cooperative way. Results show that the CSCM system was effective in providing a collaborative environment, with a higher inter-rater agreement as a result (3 times higher than the values obtained by the control group), but at the expense of 2.5 times higher average marking time. When the experimental group returned to mark in a cooperative way, the benefit of inter-rater agreement was practically lost but the time to mark was 65% lower than the control group, showing an immediate benefit of the first collaborative effort in the efficiency of the mechanics of marking.