Redesign of a Resident Evaluation Tool Using Exploratory Factor Analysis.

Abstract

Purpose: Residents at our training program identified timeliness of faculty feedback as an area for improvement, while faculty felt the evaluation tool was time-consuming and redundant. We sought to resolve these issues through stakeholder input and modern data analysis.

Methods: Core faculty and senior residents met to revamp the "Faculty Evaluation of Resident" end of rotation tool. The results of the most recent 5-years of evaluations were analyzed using Exploratory Factor Analysis, a dimension reduction tool provided in the IBM SPSS statistical package, to identify questions which were highly correlated. A new condensed tool was then generated by combining highly correlated questions with committee approval. Spearman's rank order correlation test was used to evaluate each new question versus the eliminated redundant questions. Time to survey completion and frequency of written feedback were compared using t-test. One-way ANOVA was then used to compare scores for each new question versus the eliminated questions that had been grouped together.

Results: 3,268 surveys were completed by 73 attendings regarding 55 resident subjects. Data were blinded by the program coordinator before analysis. Exploratory Factor Analysis indicated the initial 30-question instrument could be reduced to 12-questions, while retaining 96% of the variability in performance. The component matrix indicates that 4 areas accounted for the most variability in resident performance: Overall Performance, Communication, Operative Skill, and Systems Based Practice. Following implementation of the new evaluation form, attending surgeons completed resident evaluations at a median of 36 days after the rotation (IQR 18-59). Faculty left written feedback more frequently (53.4% vs 40.8%, p < 0.0001). For some new questions, the resident performances were statistically different. For example, new question 9 had an average rating of 3.55 out of 5.00, while the questions it replaced averaged 3.76-3.89 (p < 0.001). For other questions, no statistically significant difference was found. For example, new question 12 and the questions it replaced all averaged 3.75 out of 5.00 (p = 0.985). Survey response rates also improved from 35% to 76% at 2 months and 89.3% to 93.5% at 6 months.

Conclusions: Faculty input and advanced statistical analysis shortened a 30-question resident evaluation tool to 12-questions while retaining 96% of the variability in resident performance. The new instrument resulted in improved response rate and increased number of written comments from attendings. Application of Exploratory Factor Analysis to resident education represents novel use of this tool in surgical education.