Assessment by logbooks in competency-based curriculum: Is it a reliable method of objective assessment?

Abstract

Background: Logbooks have been introduced in the competency-based undergraduate medical curriculum by the National Medical Commission in India. Assessments of competencies marked in logbooks serve as a useful guide to students for skill acquired and progress. Each student should get accurate and objective feedback. Aims and Objectives: This study was planned to evaluate the effect of examiner variability on the grades awarded to learners in the logbook. Materials and Methods: This cross-sectional study involved phase 1 MBBS students. As part of their routine practical exercise, students made thin peripheral blood smears, which were graded by 3 different examiners as exceeding (“E”), meeting (“M”), or below (“B”) depending on whether a particular smear E, M, or B was the expectancy of a particular examiner. Students in Grade B were supposed to repeat the exercise. Data for grading were recorded in Excel and analyzed using statistical software R for agreement between examiners. Results: 248 phase 1 MBBS students participated in the study. The level of agreement between the three examiners was poor, meaning all examiners graded the student differently. Examiners 1, 2, and 3 were biased and inclined toward grades M, E, and B, respectively. Cramer’s V statistics confirmed the strong association (0.23, P < 0.01) of the examiner with a particular grade. A high level of agreement (0.407, P < 0.01) and strong association (0.26, P < 0.01) was found between the examiner and the grading of revision required (B) or not (E or M). Conclusion: Thus, the grading system involves examiner bias, which may be reduced by defining more objective, observable criteria for grading and reducing the number of grade categories to reduce borderline cases.