Validity Evidence for Level of Supervision Scales with Pediatric Subspecialty-Specific Entrustable Professional Activities.

Objectives: To generate validity evidence based on Messick's framework for use of supervision scales in assessment of the Entrustable Professional Activities (EPAs) specific to each pediatric subspecialty.

Methods: From 2018-2022, we asked clinical competency committees (CCC) from fellowship programs in the United States representing 14 pediatric subspecialties to submit supervision ratings for each EPA for each fellow. Program directors reported the ease of use and time required to assign ratings and familiarity of CCCs with EPAs. Internal consistency for each EPA was assessed through Cronbach's alpha. Mixed effects models were performed to estimate growth in supervision levels across fellowship years. Correlations between adjusted supervision levels and reported milestone ratings for essential subcompetencies were estimated.

Results: We collected EPA-based supervision ratings for 3,312 fellows. Most program directors reported that the scales were easy to use and took little time to assign the levels. Internal consistency for each subspecialty-specific EPA was high (alpha 0.89-0.97), as was the inter-EPA consistency within each subspecialty. There was consistent growth in supervision levels across the fellowship training years among all subspecialties. Additionally, there was a positive relationship between supervision levels and milestone-based competencies critical for the performance of the related EPAs (coefficients 0.68-0.80).

Conclusions: These data provide strong validity evidence for using the subspecialty-specific EPAs supervision scales to assess pediatric fellows. Ongoing evaluation of their use in supporting individualized learning as well as high-stake decisions will be essential.