Online peer-assessment quality control: A proactive measure, validation study, and sensitivity analysis

Abstract

Despite the generally positive learning effects of peer-assessment (PA), there are some undesirable signs and troublesome behavior needed to be addressed (e.g., biased judgments, superficial-level engagement). Anchoring on the concept of individual accountability and assessment validity, while attending to other important elements (including both subjective and objective criteria, individuals’ differentiation ability across the points on the assessment scale, and completion rate), a measure was proposed for PA quality control. This measure consists of four indexes/variables and enables the quantification and appraisal of the peer-assessor’s performance. A validation study involving two classes of undergraduate students (N = 103) engaged in the assessment of peers’ oral presentations was conducted to evaluate the effects of the proposed measure. The statistically significant results confirmed the efficacy of the proposed measure to promote better online PA performance. Additionally, to assess its applicability, sensitivity analysis targeting different aspects of our proposed measure (i.e., different plausible weighting for each of the associated indexes/variables, different correlation values between peers’ and the teacher’s ratings, and different bound values for the scale) was conducted. All significant results of the 281 test cases further substantiated the applicability and efficacy of our proposed measure under various assessment schemes for PA quality control.