Reliability and utility of anesthesiology entrustable professional activities assessed with a mobile web application

Abstract

Background

A prior study developed and validated 14 Entrustable Professional Activities (EPA) for anesthesia residents for anesthesiology residency programs to use as the basis for workplace assessment.

Objective

The goals of this study were to: 1) measure the reliability of seven of those 14 EPAs using entrustment data collected via a mobile web application (Web App) and 2) examine implementation and utility of the EPA Web App.

Methods

Web App design prioritized a simple and efficient user interface, and incorporated revisions based on feedback. Assessments were completed during the 2020–2021 academic year. Generalizability theory was used to examine the distribution of EPA global scores. Qualitative interview data of Web App usage by participants focused on domains of the Consolidated Framework for Implementation Research.

Results

The Web App was used in a total of 1116 observations in 31 residents. Two EPAs were effective in discriminating differences in resident performance, whereas two were not. With 24 observations, one of the EPAs had a G coefficient of 0.68 and phi coefficient of 0.66 approaching a level suitable for formative feedback, whereas the other EPAs did not reach the 0.7 threshold (for moderate reliability in formative assessment, not summative use) even with 24 observations. Several constructs were elicited from the interview data in the domains of intervention characteristics, individual characteristics, outer setting, and inner setting.

Conclusion

The utility of the Web App for EPA assessment was deemed positive for formative, low-stakes assessment. EPAs may not all necessarily be able to discriminate differences in resident performance. However, even a single EPA assessment may still be useful as one component of competency-based medical education via programmatic assessment.