Inter-Rater Reliability of Reflective-Writing Assessment in an Undergraduate Professionalism Course in Medical Education

Abstract

Reflective writing is increasingly being used in the teaching of professionalism. Because assessment enhances the learning process, effective evaluation of students’ reflective writing is needed. The aim of this study was to examine the inter-rater agreement between two different reflective writing assessment rubrics, which categorised reflective writings into four level of reflection, in an undergraduate medical professionalism course. The reflective writing assignments from 63 medical students enrolled in the 2017 medical professionalism course in the Faculty of Medicine Universitas Indonesia were randomly selected and independently assessed by two raters in September 2019. Intraclass correlation (ICC) analysis (two-way mixed effect, single measure) was carried out to determine the inter-rater agreement of the reflective writing assessment. The less detailed instrument showed a low ICC score of 0.43, which was classified into poor inter-rater agreement, whereas the more detailed rubric showed poor to moderate reliability, with ICC scores of 0.50, 0.50, and 0.36 for the score of each criterion, the total score of each assessed criterion, and the overall score of reflection, respectively. Utilising a more detailed (analytic) rubric to assess students’ reflective writing produced a relatively higher score of interrater reliability, although the reliability achieved using this rubric was still categorised as moderate.