Constructing a Validity Argument and Exploring Implications for the American Board of Anesthesiology's Basic Examination.

Abstract

Background: In 2014, The American Board of Anesthesiology introduced the Basic Examination as a graduation requirement for second-year anesthesiology trainees. The exam's validity has been supported by evidence demonstrating enhanced performance on other standardized exams; however, an assessment's validity is inseparable from decisions made on its behalf. This study aimed to understand the usage and implications of the Basic Exam within training programs to construct a comprehensive validity argument.

Methods: Semistructured interviews were conducted with a sample of 20 program directors from Accreditation Council for Graduate Medical Education-accredited anesthesiology training programs. Thematic analysis was performed by a 3-member team.

Results: A 56-item codebook was developed and applied to the 20 transcripts, yielding 1941 coded segments organized into 7 themes. Theme 1 highlights varied programmatic policies, including dismissal (1a). Theme 2 addresses the perceived purposes of the exam: as a tool to "weed out" residents unlikely to achieve board certification (2a), a data point supporting remediation (2b), and a distinguishing accomplishment of physician anesthesiologists (2c). Theme 3 captures programmatic implications for recruitment (3a), operations (3b), and curricula (3c). Theme 4 confirms that residents are studying for the exam, emphasizing targeted test preparation (4a). Theme 5 discusses resident implications, including stress (5a) and clinical distraction (5b). Themes 6 and 7 explore the implications of failure and equity concerns, respectively.

Conclusions: This study identifies a significantly underdeveloped validity argument supporting dismissal based on Basic Exam results and explores implications to guide future validation efforts.