Point/counterpoint: Should we stop writing and reading letters of recommendation for residency selection?

Abstract

As hospitalists involved in internal medicine and pediatrics residency selection, each of us have read letters of recommendation (LORs) like the one in Box 1. As part of the process of selecting candidates for residency training, LORs hold significant implications for both applicants and programs and are tainted by deep-rooted flaws. These defects continue largely because of our collective failure to confront and address the glaring issues within the process. At best, LORs offer marginal benefit in selecting residents; at worst, they become overt channels for bias, inequity, inequality, and arbitrariness, often devolving into exercises of inanity, untruthfulness, obfuscation, and even propaganda. As such, they decrease the integrity and purpose of the residency selection process.

For example, gender bias in residency application LORs has been noted for Radiology, Orthopedic Surgery, Female Pelvic Medicine and Reconstructive Surgery, Cardiovascular Surgery, Emergency Medicine, Pediatrics, Anesthesiology, Radiation Oncology, Ophthalmology, Internal Medicine, and General Surgery, among others.¹ Women often find themselves described in these letters with communal traits, such as helpful and caring.² In contrast, men are more likely to be portrayed with agentic traits, including leader and taking initiative. LORs for women also tend to focus more on personal appearance (such as the misogynistic but real-life example about Ms. J. Smiths' figure in the above letter) and personal life. They also contain more doubt raisers (e.g., “it appears her health and personal life are stable”²), including hesitancy from the recommender, use of faint praise, potentially negative comments, unexplained comments, and irrelevancies.²

Ethnic and racial biases are also prominent in residency LORs, where differences in language can subtly influence readers' perceptions of candidates.³ As with gender, agentic and communal terms are used differently based on a candidate's ethnicity or race. Even apart from bias, LORs compound inequity. The process of obtaining LORs favors already advantaged groups who are more likely to have access to the most influential letter writers. Students often spend an inordinate amount of time searching for the “right” letter writer, often choosing those with titles or positions of power over those who know them best. LORs also tend to focus only on positive aspects of applicants, neglecting the comprehensive portrayal of a candidate's journey, struggles, and growth. This one-sided representation undermines the principle of holistic review (a balanced assessment of an applicant's experiences, attributes, and academic metrics⁴) by not fully acknowledging the resilience and perseverance shown in overcoming challenges, especially among disadvantaged applicants. Even worse, despite the purported value of holistic review, residency program directors (PDs) often view the demonstration of improvement or overcoming personal setbacks negatively and perceive narratives about growth as coded language for deficits.⁵

Finally, the interpretation of LORs varies significantly among readers. Studies suggest that readers are not able to discern from letters alone who the top performers are.⁶ In addition, the recent rise in plagiarism and potential use of artificial intelligence (AI) for generating LORs further undermines their credibility.⁷ An increasing number of LORs are being produced by generative AI, and readers are unable to reliably differentiate between human- and AI-authored versions.⁸ For all the reasons discussed above, it is not surprising that LORs have been shown to be poor predictors of residency performance.^{6, 9}

Efforts to improve the process of writing LORs in various medical specialties have been undertaken, primarily through the introduction of standardized letters (SLORs). SLORs employ a uniform format designed to provide consistent and comparative information across all applicants. However, this approach predominantly relies on normative comparisons, where writers rank candidates based on flawed and incomplete data. This leads to grade inflation: in one study of otolaryngology residents, all 10 SLOR attributes for all candidates had a mean above the 80th percentile.¹⁰ Moreover, persistent issues such as gender bias, racial bias, and a general lack of validity evidence continue to mar SLOR effectiveness.^{11, 12}

Some argue that occasionally accurate LORs are more compelling than the substantial evidence of their deep flaws. We believe this mindset is a manifestation of common cognitive biases in human reasoning. These include confirmation bias, framing effect, base-rate neglect, visceral bias, Semmelweis reflex, hindsight bias, and premature closure.¹³ As readers of LORs, we often believe in our own inherent ability to “read between the lines” and “determine the truth” but we attempt both at our own risk. Despite numerous workshops, papers, and initiatives aimed at improving letter-writing skills, it is unrealistic to expect significant behavioral changes from the vast number of letter writers and readers involved in residency selection. The reluctance to recognize the inherent flaws and unfairness in LORs is likely because doing so would lead to the inevitable conclusion: we should cease writing and relying on them.

In reflecting on the evolution of the residency selection process, it is crucial to consider the historical context of assessment in medical education. There was a time when assessment amounted to little more than a cursory checkmark exercise, leaving graduate medical education (GME) with little faith in the integrity or quality assurance of graduates emerging from undergraduate medical education (UME). This lack of trust in assessment data led to a reliance on LORs from trusted and respected colleagues. The ethos of “big name” letter writers became a significant factor, compensating for untrustworthy assessment data. However, with the advent of Competency-Based Medical Education (CBME), the assessment landscape has dramatically transformed. Though UME assessment continues to have shortcomings, CBME has undeniably improved the process, offering a more reliable and equitable evaluation of applicants. CBME marks a significant advancement over LORs, now an obsolete tool with the advent of more sophisticated and trustworthy assessment methods.

In clinical practice, we uphold the two principles of using evidence to guide decisions and monitoring biases to minimize harm. This ethos should extend to residency selection. Assessment, often used to safeguard societal interests, must be underpinned by credible evidence for the decisions made. LORs fall short in this regard, lacking the necessary evidence to substantiate their validity. If LORs were a medical procedure, they would not gain approval from regulatory bodies. This disparity highlights the urgent need to reevaluate and align residency application assessments, including LORs, with evidence-based standards.

LORs serve as channels for bias and inequity, favor well-connected applicants, focus on selected positive attributes at the expense of true holistic assessment, and have little to no validity at predicting performance in residency. We should stop writing and reading LORs for residency selection, now.

LORs are far from perfect but calling to remove them based on inequity, overemphasis on positive attributes, and poor validity evidence is like sweeping one leaf in a forest. In a systematic review, Lipman et al. summarize the literature on metrics used for resident recruitment. Study after study shows that grades, standardized test scores, additional degrees, interviews, Medical Student Performance Evaluations (MSPEs), personal statements, honors/awards, and more are all compromised by bias, as well as the potential for cheating and poor predictive validity.¹⁴ Calling for the removal of LORs oversimplifies a complex discussion and unnecessarily singles out one part of the residency selection process. LORs are not the problem, they are a symptom of a broken system.

LORs are flawed, but one of the main suggestions to increase diversity, mitigate bias, and increase credible decision making in residency selection is holistic review.^{15, 16} The very idea of holistic review is that each piece of data is imperfect, and only through the systematic review of all data with a diversity of opinions (i.e., groups) can we begin to minimize bias in residency selection.⁴ So, will one less piece of data really lead us to less bias, or will it just shift our emphasis onto another piece of biased data? The argument to stop writing and reading LORs on the grounds of bias ignores the fact that all metrics and the entire process of residency selection are compromised by bias.

Contrary to what the Point authors state, LORs might be a tool to increase equity. The Point authors have made the argument that LORs compound inequity since all applicants do not have the same access to letter writers. This is an example of equality, and we agree that equality is neither possible nor desirable. Equity seeks to get each person what they need, in hopes of reaching an equal outcome. With this framing in mind, LORs may provide a unique opportunity to promote equity in the residency selection process. LORs are a key tool that faculty can use to advocate for students and lift up those that are marginalized or underrepresented.^{17, 18} Individual faculty have limited control over grades, awards, or the specific opportunities a student may have. But as educators, we can promote applicants in unique ways, describing their passions, interests, and challenges they have overcome in a way they may not be able to highlight for themselves. Given the number of applicants, most experience the residency selection process as high stakes and impersonal, but a LOR is one of the few opportunities for students to connect with faculty. Removing LORs may improve equality, but it will eliminate one of the ways we can promote equity in a selection process where it is currently lacking.¹⁹

Empirical data show residencies really value LORs and put them to good use. In the biennial survey conducted by the National Resident Matching Program, GME programs consistently cite LORs as a main factor in choosing applicants to interview (80%–90% of the time).²⁰ In fact, LORs ranked higher in importance than standardized tests and personal statements. This is amplified in smaller specialties like Dermatology, Vascular Surgery, and Urology where LORs are almost unanimously perceived as important.²⁰ Similar trends are seen in fellowship applications where LORs have magnified significance.²¹

The Point authors have stated that one shortcoming of LORs is their failure to comprehensively portray all the struggles, growth, and journey of an individual applicant. Setting aside whether this is even a reasonable expectation for one faculty member to comment upon, they go on to acknowledge that PDs may penalize an applicant when there is language in the LOR about growth or improvement. If the Point authors feel a transparent portrayal of each applicant's journey is lacking, we would point them to the MSPE rather than the LORs.²² Regardless, removing LORs will not make for a more transparent residency selection process.

If LORs are useless, then why are so many continuing to use them? In a study of 150 Emergency Medicine PDs, only one advocated for removing LORs from residency selection. When asked for the most important characteristics in choosing who to interview, 139 ranked the LOR first.²³ In another study of Anesthesia PDs, most agreed there is value in using LORs to choose who to interview and to look for important keywords and phrases.²⁴ In a national survey of Pediatric PDs, commonly used phrases and keywords in LORs were found to be interpreted in a consistent manner. Importantly, this study found that almost 90% of PDs would consider a weaker candidate more favorably if they had a well-crafted LOR, once again underscoring the opportunity of using LORs to promote applicants.¹⁸ Clearly, LORs continue to be used because some key decision makers see potential value. Their depiction as rampant sources of bias should be interpreted with caution. In a 2023 systematic review on residency selection, the authors concluded that the case for bias in LORs is mixed and there is some data to support their predictive value. This led them to conclude that there is more evidence to continue using LORs, while USMLE scores, grades, national school ranking, additional degrees, and receipt of awards should have a limited role.¹⁴ Maybe we have chosen the wrong metric to debate.

The Point authors would like you to believe that assessment in medical education has evolved with the arrival of CBME, but if it is all tainted by bias, has it really evolved?²⁵ CBME was raised in the Point as a solution, rendering LORs as obsolete since we now have reliable, accurate, and trustworthy assessments. This could not feel further from reality in UME, where CBME is challenging to implement, normative assessments still dominate, and students are oriented primarily toward hiding their weaknesses to try and set themselves apart.²⁶ CBME, as currently implemented, is not the solution. In fact, like holistic review, CBME is built on the idea that utilizing many flawed and imperfect assessments will allow for a more complete picture of a trainee's development.²⁷ Therefore, should not their argument to remove LORs also extend to other forms of assessment that CBME holds with high esteem? If the Point authors want to remove any biased and flawed data from the residency selection process, this slippery slope leads to one solution: residents matched by a lottery. Does that feel extreme? The Netherlands have tried a lottery, stopped it, and are now bringing it back.²⁸

The Point authors dismissed SLORs as a potential improvement, citing reasons such as bias and normative comparisons. However, there might be more to the story. In some contexts, the SLOR was more reliably interpreted and reduced the time that residency programs needed to review LORs.²⁹ In a study of Pediatric applicants collecting validity evidence for SLORs, they found it to be moderately reliable, correlate to admission decisions, and differentiate among applicants even though faculty tended to inflate their ratings on the scale.³⁰ The amount of variance attributed to the applicant in this study (i.e., ability for SLOR to differentiate between applicants) is much higher than most assessments found in medical education. Is not this the kind of validity evidence that the Point authors have called for? In fact, building upon the validity evidence, a decision study showed that by reading four SLORs, one could reliably differentiate between applicants.³⁰ This is critical since residency programs need to complete a final rank of all applicants. Evidence that any piece of information will predict success in residency is lacking, but utilizing SLORs seems to be one way to improve residency selection, mitigate bias, and provide programs with the data they desire.³¹

Finally, the Point authors have beseeched readers to take an evidence-based approach to guide decisions in residency selection. Yet study after study has shown that no piece of data in the entire residency selection process seems to predict future performance.^{14, 31} LORs are consistently reported as valuable, with the potential to promote equity, while simultaneously being just as flawed as any other metric in the residency selection process. Removing LORs undermines the very idea of holistic review; we believe pulling one thread (i.e., LORs) from the tapestry (i.e., residency selection) has the potential to do more harm than good.

The Counterpoint authors have asked: if the entire residency application process is flawed, why then focus solely on letters? Choosing LORs to remove first is not arbitrary. Rather, it is a strategic move to tackle a classic case of normalized deviance within the residency selection process. Normalized deviance, a phenomenon where deviant practices gradually become accepted as normal within an organization, often leads to a lowered standard of ethics and performance.³² To those entrenched in the system, these practices seem routine and acceptable, while they appear problematic to outsiders. In the case of residency selection, LORs are a prominent example of this deviance. They have become a routine part of the process, despite their inherent flaws and lack of fairness. The first step to ending normalized deviance is to acknowledge and make the problem visible. Removing LORs would do this in an instant. Once this step is taken, the focus can then shift to other aspects of the residency application and selection process. The goal is to create a system that is fair, equitable, accurate, valid, and valuable, rectifying not just a single flawed aspect but challenging a pattern of normalized deviance that has been accepted for too long. This approach is not just about removing a single problematic element; it is about taking a stand for greater integrity and effectiveness.

Medical education assessment has evolved. CBME's narrative assessments are shared with a wide array of stakeholders including trainees, competency committees, PDs, and institutions, ensuring transparency and collective scrutiny. In contrast, LORs remain limited in visibility, accessible only to the authors and a select few reviewers. In their present form, they are anathema to CBME: isolated high-stakes assessments based on limited data with low-quality validity evidence. As such, all biases, inaccuracies, and inequities are heightened. Medical educators are still learning how to use CBME to clearly define and assist medical students in meeting criteria essential for graduation. These data, collected from many sources, should primarily facilitate formative assessments and feedback, rather than summative judgments.³³ Once graduates meet these criteria, medical schools can confidently assert their readiness for residency, backed by concrete validity evidence. LORs would no longer be needed. We do not need to wait for this. LORs are causing harm now and we should stop writing and reading them for residency selection.

The authors declare no conflict of interest.