Board scores in the spotlight: Public reporting and the unintended consequences

Abstract

The American Board of Emergency Medicine (ABEM) recently announced plans to publicly report program-level board certification examination pass rates.¹ This initiative will present program-level board pass rates for public viewing. Multiple arguments can be made for such a change. Medical education is increasingly seen as a service for which trainees have paid large sums of money and sacrifice a significant amount of time and effort. Transparency and accountability to learners that show services are high quality is important. Additionally, public reporting of program board pass rates provides a mechanism for accountability to society at large and aligns with ABEM's mission “To ensure the highest standards in the specialty of Emergency Medicine.”² Graduate medical education (GME) is largely funded through tax dollars, and an argument can be made that the public should be able to see the degree to which different training programs are helping their graduates pass certifying examinations.³ Finally, this change would align ABEM with multiple other major American Board of Medical Specialties (ABMS) member boards who publicly report program pass rates, such as the American Board of Internal Medicine, American Board of Pediatrics, and American Board of Family Medicine.⁴ These rationales have merit, but unintended consequences lurk around the corner. In this perspective, we describe the potential negative impact of publicly reporting program-level certifying examination pass rates. Specifically, we explore how public reporting could disincentivize holistic review of applicants during residency recruitment. We propose actionable strategies that various stakeholders may consider for balancing transparency with the broader mission of holistic review and inclusive recruitment practices in GME.

Humans are influenced by incentive structures. As rational actors, we tend to (either implicitly or explicitly) alter our thinking and behavior when different incentive structures are presented to us. When such structures lead to unwanted consequences, they are labeled “perverse incentives.” A colloquial term for the impact of a perverse incentive is sometimes called the Cobra Effect.⁵ The Cobra Effect draws from an anecdote in which a governmental effort to reduce the number of cobras in Delhi, India, backfired. The initiative offered a bounty for dead cobras, so people began breeding cobras to turn in more dead snakes. The bounty was intended to incentivize the killing of cobras, hence decreasing the overall population. However, people quickly realized they could game the system for financial gain, leading to an overall increase in the cobra population. The incentive structure led to unwanted consequences that could potentially have been predicted by considering how people would respond.

We believe that publicly reporting program certifying examination pass rates may lead to its own Cobra Effect, potentially altering program incentives in ways that undermine holistic recruitment efforts. Holistic review in residency selection has gradually shifted programs away from an overreliance on academic metrics like test scores and class rank in favor of a broader (i.e., more holistic) evaluation of an applicant's strengths, interests, and value alignment with a residency.⁶ While academic metrics still play a role, they are no longer the sole focus, thanks to growing awareness of their limitations. Many traditional academic metrics lack validity evidence for use in GME recruitment and do not predict future residency performance.^7-9 Additionally, concerns about inequity and bias in these metrics—such as the systematic disadvantages faced by minoritized groups—have highlighted the need for more inclusive strategies.^{10, 11} Holistic review, while not a standardized approach, is recommended by most national organizations (AAMC, AMA, ACGME) because of its potential to improve equity in the residency selection process.^12-16 The competency-based medical education (CBME) movement has further underscored that knowledge and test-taking ability are only parts of a broader framework of competence.¹⁷ We fear that publicly reporting program board pass rates for the sake of transparency and accountability may underestimate the looming Cobra Effect of incentivizing programs to return to overemphasizing test scores when evaluating applicants.

To be clear, we are not suggesting that passing the certifying examination is unimportant. When compared to individuals who are not board certified, board certification has been linked to meaningful patient care outcomes in future practice.^18-21 Thus, the certifying examination pass rate presents a metric that has some validity evidence to support its use in assessing individual graduates’ readiness for practice. The key issue lies in differentiating whether the residency program's quality or the individual's test-taking abilities has more influence on passing or failing the examination. The rationale for sharing program pass rates relies on a false premise: that all programs recruit learners who enter residency with a similar chance of passing the certifying examination. However, it is very unlikely that testing ability and medical knowledge is equally among matriculating cohorts across programs. The best predictor of passing a standardized certifying examination is performance on previous standardized examinations.^22-31 In contrast, the evidence is much less clear about whether the quality of residency training is associated with board passage rates.^32-40 Use of program-level board pass metrics as a signal of residency training quality fails to account for inherent differences in programs’ incoming residents' prior standardized examination performance, making intraprogram comparisons potentially misleading.

The public reporting of board certification pass rates will likely exert significant pressure on program directors (PDs) to mitigate the risk of examination failures. This is a good goal to aspire toward; PDs should help their graduates pass the certifying examination. However, they may feel pressured to avoid failures not only by improving education and training, but also by changing recruiting practices. Even a small number of individual examination failures could greatly alter a program's overall pass rates given the relatively low number of graduates per year from ABEM training programs. Programs might fear risk of impaired recruiting should applicants use program board pass rates as a metric to evaluate and rank programs. The most straightforward way for PDs to minimize risk is to recruit applicants with the least risk of examination failure. This aligns with Campbell's Law, which suggests that when any measure is emphasized as a marker of social standing (e.g., high pass rates equates to a better residency training program), it will be prone to distortion or corruption.⁴¹ In this context, program board pass rates taking on such a visible and elevated status will no doubt make prior testing performance a more important metric for PDs to consider when making decisions on whom to invite for an interview and how to rank them on final lists. PDs, as rational actors, may make decisions to “protect” their programs as well as their personal reputation by prioritizing metrics that maximize pass rates and minimize examination failures. If programs focus significantly on recruiting applicants with proven test-taking abilities, the entire system shifts toward this strategy. In turn, this could lead to students and schools responding by placing even more emphasis on testing metrics. If a residency program chooses to resist this shift on principle, maintaining a more holistic review, they are opening themselves up to scrutiny from the public and applicants if their pass rates begin to lag behind other programs. Ultimately, a cycle of competitive defensiveness is perpetuated over time.

Public reporting of program board pass rates risks undermining national momentum made toward programs using holistic review for applicant selection. For many programs, the added pressure of transparency may act as a disincentive to invest in resource-intensive, nuanced holistic review processes. Instead, it may nudge programs—if not outright shove them—back toward relying heavily on easily sortable academic metrics like test scores. Programs that are already stretched thin and searching for simpler application review strategies may find this shift particularly appealing. Holistic review has led to an emphasis on increasing the number of diverse residents in programs nationwide. A return to prior selection strategies has potential to undermine the diverse representation of physicians, which we know has a positive effect on patient outcomes.^42-44 Public reporting could inadvertently set back holistic review efforts, prioritizing expediency over the broader, equity-driven goals that have shaped medical education reform in recent years.

Medical education has made great strides toward holistic recruitment strategies. The potential unintended consequences highlighted by Campbell's Law and the Cobra Effect reveal the risk of incentivizing programs to return to an overemphasis on test-taking abilities in resident selection. More research is needed to explore the complex alchemy of how certifying examination results can be transparently reported for stakeholders without disincentivizing holistic recruitment and selection practices. While we encourage residency programs to share program board pass rates with prospective applicants during the recruitment process when there is an opportunity to provide appropriate context and nuanced discussion, we recommend that ABEM not make board pass rates public.

The authors declare no conflicts of interest.