Application of a Validated Assessment Tool to Compare Autonomy and Operative Performance Between Surgical Residency Programs

Abstract

PURPOSE

Every residency has the responsibility to determine the competency of its residents and their readiness to practice after graduation, but few objective tools exist to assess residents. The Zwisch grading scale is a validated assessment tool for operative performance which has been incorporated into a smartphone application, called System for Improving and Measuring Procedural Learning (SIMPL). This application allows residents and attendings to evaluate a resident's performance in real time. Since its implementation there have been no studies that have compared scores across all participating general surgery programs in order to identify if varying procedure difficulty affect supervision and performance ratings amongst programs overall.

METHODS

The SIMPL database was queried for attending ratings for Post Graduate Year (PGY 1-5). Meaningful autonomy (MA) was defined as passive help or supervision only and satisfactory performance (SP) was defined as practice ready or exceptional. Procedures were classified as “core” or “advanced” based on the American Board of Surgery Surgical Council on Resident Education (SCORE) criteria. Work relative value units (wRVU) obtained from the Centers for Medicare & Medicaid services data were matched to procedures and categorized into 3 groups (<13.07, 13.07-22, >22) based on previous work. Proportions of advanced and high wRVU (>22) procedures were calculated for each program and PGY and matched back to each corresponding evaluation as potential proxies to resident exposure to more complex operations. All evaluations were divided into quintiles based on the corresponding program's proportion of advanced and high wRVU procedures. Data were summarized using descriptive statistics and generalized estimating equations (GEE) logistic regression models were used to assess whether program proportions of advanced or high wRVU procedures are predictive of MA or SP overall and among PGY 5 residents.

RESULTS

Overall, proportions across programs ranged from 2.2% to 32.5% (mean 12.4%) and 2.4%-22.9% (mean 11.9%) for advanced or high wRVU procedures respectively. Across PGY levels, the mean proportion of advanced (7.6, 9.6, 13.1, 13.4, 14.2) and high wRVU (4.4, 6.5, 9.8, 15.3, 19.8) procedures increased with each step in PGY. In descriptive analyses of the entire group, lower numbers of MA and SP were noted with increasing proportions of advanced/high RVU. The relationship was similar for all core procedures and when limited to PGY 5 residents, however it was not entirely consistent and showed no association with advanced procedures. Similarly, in the models lower adjusted odds of MA and SP were noted for evaluations from programs with higher proportions of advanced or high wRVU procedures for the overall group but this was less consistent when limited to PGY 5 residents and even more variable when limited to advanced procedures (Table 1).

CONCLUSIONS

There is wide variation in the proportion of advanced or high wRVU procedures reported at different programs. Overall MA and SP tended to be lower for programs reporting more advanced procedures overall and equivocal in programs with higher wRVU procedures. Programs with higher percentage of advanced procedures showed an increase in MA and SP when looking at performance as a PGY5 during core procedures only. The MA and SP of PGY5s performance during advanced procedures was not changed by an increase in exposure to advanced procedures. These results indicate that programs with more advanced cases may not necessarily train surgeons with the appropriate level of skill, knowledge or confidence needed to be an independant and safe surgeon.