Assessment burden by design: exploring the variability in competence by design assessment forms.

Abstract

Background: The Royal College of Physicians and Surgeons of Canada's (RCPSC) Competence by Design (CBD) framework has been criticized for increasing assessment burden due to the high number of required Entrustable Professional Activity (EPA) assessments. Another contributing factor may be the inefficient design of assessment forms. We explored variability in form design to identify differences that could impact learners' and assessors' experience with CBD.

Methods: Annotated assessment forms for the 'Core 1' (C1) EPA were collected in March 2023 from all (n = 14, 100%) RCPSC emergency medicine (EM) residency programs in Canada that had implemented CBD. Forms were divided into six sections to compare their design. The variability between form sections was described relative to RCPSC recommendations on form design.

Results: EPA assessments were completed within six learning management systems. Variability was found throughout the form including the number of context variables, included milestones, milestone rating criteria, and text boxes for narrative feedback; phrasing of narrative feedback prompts, milestone descriptions, and entrustment score criterion; visual presentation of the entrustment score; arrangement of form components; and the components' selection format. The mandatory completion of form components was inconsistent. Some forms could be partially completed by residents. One form added a global performance rating scale. The number of clicks required to complete a form ranged from 12 to 47.

Conclusion: We found considerable variability in the design of the EM C1 EPA assessment form. Variations that make completion more challenging could increase assessment burden. CBD programs should be aware of this and seek to optimize the design of their forms.