Intra-rater reliability of in-person versus simulated remote synchronous faculty evaluation of pharmacy student objective structured clinical examinations

Abstract

Introduction

The utilization of objective structured clinical examinations (OSCEs) provides a vital learning and assessment opportunity within health professions education. While technology has been utilized, there is a lack of comparison of in-person versus remote synchronous OSCE evaluation. The objective of this study was to evaluate the intra-rater reliability of faculty evaluation in-person versus simulated remote synchronous evaluation of pharmacy student OSCE performance.

Methods

For three academic years, pharmacy student OSCE performances in two professional years at one institution were recorded while the instructors evaluated the students in-person. Following a six-month washout, instructors evaluated the same students remotely simulating synchronous evaluation using the same standardized rubric. Intra-class correlation coefficients (ICC) were calculated to determine the evaluator intra-rater reliability. Overall and individual OSCE station scores, the need to remediate, and overall course grades were compared between the cohorts.

Results

A total of 157 out of 380 OSCE activities were evaluated both in-person and remotely by 11 faculty members. The ICC scores were 0.9 overall and 0.9 for each individual year's OSCE station. There was a difference in overall OSCE scores based on simulated remote synchronous versus in-person assessment (84.0 % vs. 85.0 %) and for the individual P3 OSCE station (84.5 % vs. 86.0 %); however, there was no difference in the number of students requiring remediation or overall course grades.

Conclusion

This study demonstrated excellent intra-rater reliability between in-person and remote synchronous evaluations of an in-person pharmacy student OSCE. The study findings suggest remote synchronous evaluation is a viable alternative to in-person evaluation of OSCEs.