Influence of Evaluator Expertise on Inter-Rater Reliability in Crown Preparation Assessment Without Formal Calibration.

Abstract

Aim: To investigate the relationship between evaluator expertise and inter-rater reliability during crown preparation assessment, as well as whether students' academic level affects performance outcomes.

Methods: This research used a two-phase design. Initially, it enrolled 92 third-year and 90 fourth-year students; after subjecting the cohorts to predetermined inclusion criteria, 85 students of each cohort were retained to be included in the study. In Phase 1, the third-year students who prepared a metal ceramic crown on a typodont tooth were evaluated by two senior and two junior assessors using an analytic rubric that was validated on a 10-point scale. In Phase 2, the fourth-year students did the same task and it was only assessed by senior evaluators. The intraclass correlation coefficients (ICC) were used to analyse the inter-rater reliability, and t-tests were used to analyse differences in groups.

Results: In Phase 1 the reliability of all the evaluators was low (ICC = 0.035). Within this phase senior evaluators showed moderate reliability in the absence of formal calibration (average-measure ICC = 0.634), and junior evaluators demonstrated systematic disagreement (ICC = -0.074). In Phase 2, senior evaluators demonstrated good-to-excellent agreement. Third- and fourth-year performance did not differ significantly (mean difference = 0.28; p = 0.255; d = 0.18).

Conclusions: Within Phase 1, inter-rater reliability differed descriptively between evaluator groups, with senior evaluators demonstrating more consistent scoring patterns than junior evaluators in the absence of formal calibration. There was no statistically significant difference observed in academic levels in this cohort.