Reliability and Validity of the Respect Factor in Student Evaluations of Clinical Educators.

Abstract

Purpose: Medical student mistreatment has profound negative effects on student education. When medical students feel mistreated, they may also feel that they have been treated disrespectfully. This study examines the validity and reliability of a tool to measure students' perception of educators' level of respect.

Method: Data from 516 student raters of 2,534 clinical educators (i.e., faculty and residents across 8 clinical departments) were used to investigate validity evidence based on Kane's validity framework for an instrument that includes 2 items for measuring educator respect and 7 items for measuring teaching effectiveness. Exploratory and confirmatory factor analyses were conducted to investigate construct validity. Generalizability theory analysis was conducted to project estimates of the level of reliability of departmental-level respect scores for different data collection scenarios. The raters were third- and fourth-year students attending clinical rotations at Sidney Kimmel Medical College, Thomas Jefferson University, during the 2022 to 2023 academic year.

Results: Exploratory factor analysis revealed 2 correlated latent factors that represent respect and teaching effectiveness. The estimated confirmatory factor analysis model, with 2 first-order latent factors (i.e., respect and teaching effectiveness) and 1 second-order latent factor (i.e., teaching quality), resulted in a root mean square error of approximation index of 0.10 (indicating mediocre fit), a nonnormed fit index of 0.95 (indicating good fit), and a standardized root mean square residual of 0.03 (indicating good fit). Generalizability theory decision studies revealed plausible scenarios that would lead to reliability estimates between 0.71 and 0.81.

Conclusions: This study suggests that the respect rating scale yields sufficiently valid measures of students' experiences and reliable department-level respect scores for plausible scenarios when ratings are obtained from 10 students nested in each of 35 educators per department. The results also suggest that the scale allows for valid decision-making about feedback to clinical educators and their departments.