Validity: Conceptualizations for anatomy and health professions educators

Abstract

Validity, a cornerstone of assessment theory and practice, holds particular importance in the context of health professions education. In disciplines such as anatomy education, where assessments often bear significant consequences for learners and their future roles in healthcare, the integrity and applicability of testing instruments are paramount. Validity evidence is essential for three primary reasons: (1) it provides support for the intended purpose of the assessment, (2) it conveys to stakeholders that the results of the assessment are both credible and meaningful, and (3) it guides test development and refinement towards the goal of ensuring fairness, improved decision-making, and protection against the misuse of instruments. Without validity evidence, the results of an assessment may be misleading, misinterpreted, or improperly applied, potentially leading to incorrect conclusions or decisions about a student's progress.

This editorial explores the evolution of the concept of validity, discusses contemporary perspectives of validity, and discusses validity's critical role in anatomical and health professions education (HPE) assessment. By examining validity's historical development, contemporary perspectives, and practical implications, educators can better understand how to design, analyze, and refine assessment and evaluation tools that meet the rigorous demands of HPE.

The concept of validity has evolved significantly in the nearly eight decades since its emergence. The earliest recorded conceptualizations of validity surfaced in the late 1940s¹ and early 1950s,² in a series of technical reports authored by the American Psychological Association (APA).^{2, 3} In the first technical report, four types of validity were identified: predictive, content, congruent, and status.² This report was quickly followed by a second technical report, in which four different types of validity were described: construct, content, predictive, and concurrent³ (For definitions of these, and select other early types or aspects of validity that have been proposed, see Table 1).

This second technical report would lay the groundwork for what would become known as “The Standards”; a set of recommendations prepared and revised by the Joint Committee on Standards for Educational and Psychological Testing of the American Educational Research Association, the APA, and the National Council on Measurement in Education (NCME), as guidelines for the evaluation, development, and use of testing instruments.⁴

The first edition of the Standards, published as “The Standards for Educational and Psychological Tests and Manuals”,⁴ introduced “criterion-related” validity, encompassing predictive and concurrent validity as subcomponents, leading to a three-part validity structure.⁴ This three-part validity structure, consisting of construct, criterion, and content validity, would commonly become known as the Validty Trinity. The Validity Trinity, or “Three C's” of validity, represents the “classical” view of validity,⁵ which was held widely from 1966 through the late 1980s⁶ (For a timeline of the evolution of our societal understanding of validity, see Figure 1).

The Validity Trinity was challenged in 1989 by Samuel Messick, who proposed that the Validity Trinity was fragmented and incomplete, as it failed to take into account both the evidence of the value implications of test scores as a basis for action and the social consequences of score use.^{7, 8} Consequently, Messick presented the unified validity framework, which redefined the concept of validity in testing.⁹ This framework emphasized that validity is not merely a collection of separate and substitutable types but a unitary construct centered around the concept of construct validity.⁷ Messick emphasized that construct validity should encompass all forms of validity evidence, making it the central focus of measurement, which may occur quantitatively and/or qualitatively in a variety of forms (e.g., test scores, performance/skill assessments, surveys, observations, interviews, and content analyses).¹⁰ To address the complexity of validity as a unified concept, Messick proposed six interdependent and complementary sources of validity evidence.¹¹ The six sources were later condensed to five when the unified theory of validity replaced the Validity Trinity in the 1999 revision to The Standards.¹²

Specifically, The Standards outline the following sources of evidence to support an instrument's validity: evidence that is based on test content, response processes, internal structure, relations to other variables, and finally, evidence based on the consequences of testing¹³ (See Table 2 for descriptions of each source of validity evidence outlined in The Standards; for examples and elaboration, please also see Beckman et al.¹⁴).

Taken together, these sources of evidence function as general validity criteria for all educational and psychological measurements, including performance assessments.^{7-9, 13, 15-17} Thus, in this conceptualization, validity may be operationally defined as the degree to which an instrument accurately measures what it is intended to measure, meaning the results can be confidently interpreted as reflecting the intended skills, knowledge, or attitudes being assessed.¹³

In HPE, validity is often invoked to assert the quality of an assessment instrument or tool and justify its use in high-stakes decisions, such as admissions and fitness for practice. This reliance on validity, sometimes referred to as a “god term”,^{18, 19} is used rhetorically to convey adherence to high standards.¹⁹ However, there is notable inconsistency in how validity is conceptualized and applied, as highlighted by recent reviews.^{20, 21} While this variability is often attributed to a lack of understanding of modern validity theories, it may also reflect the diverse conceptualizations of validity that serve different purposes in HPE, a field influenced by a range of disciplines, including educational psychology, sociology, and measurement science.¹⁹

In 2017, St-Onge et al. identified and articulated three distinct yet coexisting conceptualizations of validity within the HPE literature: validity as a test characteristic, validity as an argument-based evidentiary chain, and validity as a social imperative.¹⁹

Validity as a Test Characteristic considers validity an intrinsic quality of an assessment instrument or tool. From this perspective, an instrument that has been validated in one context is assumed to maintain that validity universally across similar contexts. This approach offers practical utility by suggesting that ‘validated’ tools provide a “gold seal” of instrument quality, which can be particularly appealing for those seeking quick, reliable assessment instruments and who have no desire to frequently collect new data to support an instrument's use in different contexts.¹⁹

Those adopting this conceptualization view validity as an inherent property of the tool itself, independent of the specific content or context in which it is used. One adopting this conceptualization of validity may seek to answer the question, “Does this instrument or tool accurately measure what it claims to measure?” However, individuals adopting this conceptualization should be cautious, as the treatment of validity as an unchanging characteristic of the instrument can create a “false sense of security,” when applied in different contexts or populations.¹⁹ This is particularly concerning when instruments are utilized in different contexts and with different intentions beyond the original area of study. For instance, Gould (1996) highlights the risks of misusing “validated” tests in The Mismeasure of Man, showing how IQ tests, intended to measure intelligence with a single score, were applied in inappropriate contexts—such as immigration—to wrongly categorize entire ethnic groups as less intelligent.²²

Validity as an Argument-Based Evidentiary Chain emphasizes a more dynamic process, where evidence must continually support the interpretations and uses of assessment instrument results. In this view, validation is ongoing and context-dependent, focusing on how an assessment instrument is used rather than its inherent qualities. This discourse aligns well with the scientific method, requiring an accumulation of evidence for each new assessment scenario, thus fostering rigorous examination and flexibility in test interpretation.¹⁹ Kane's framework supports this notion of validity as an Argument-Based Evidentiary Chain, which emphasizes that validity is established through a structured argument, where evidence is gathered to support specific claims about test score interpretations and uses.²³ His framework outlines four stages—scoring, generalization, extrapolation, and interpretation/use—each requiring distinct evidence to ensure the assessment instrument's results are reliable, representative, and appropriately applied. This model also aligns with Messick's unified theory by emphasizing that validity is not inherent in the test itself but is demonstrated through the evidence supporting the interpretation and use of scores.

In this conceptualization, validity is not tied to the tool but rather to interpreting its results, relying on continuous evidence collection and analysis. An individual adopting this conceptualization of validity may seek to answer the question: “Am I drawing valid inferences from these test scores?” or “Are my interpretations of these test scores valid?”

Validity as a Social Imperative broadens the concept of validity beyond the test and its interpretations to include the societal and ethical impacts of assessment. This view stresses that assessments should serve to measure and support societal needs and ethical considerations, such as equity and the holistic preparation of healthcare providers. Validity, from this perspective, thus emphasizes the broader consequences of testing on learners and society, highlighting the importance of fair and responsible assessment practices.¹⁹ It expands the traditional technical perspective to include societal impact, recognizing that validity is about accurate measurement and ensuring that assessments contribute to a just and inclusive society. The notion of validity as a social imperative is supported by the Explanation-Focused View of Validity,²⁴ which aims to embed fairness, equity, and ethical considerations into its framework. The Explanation-focused View of Validity, like the Argument-Based Evidentiary Chain conceptualization, defines test validity as the process of understanding and explaining variations in test scores, but emphasizes the contextual, ecological, and social factors that influence performance. This approach extends beyond the psychometric properties of the instrument or tool to encompass broader concerns about fairness, equity, and the real-world implications of assessments.²⁴

The ‘Validity as Social Imperative’ conceptualization emphasizes assessments' broader societal and individual impacts, highlighting the ethical responsibility of ensuring positive and equitable outcomes from assessment practices. An individual adopting this conceptualization of validity may seek to answer the question: “What are the consequences or impacts of this assessment for the individual learner and for society?” While the “Validity as Argument Based Evidentiary Chain” framework stands in direct opposition to the “Validity as Test Characteristic” framework, it stands as a strong complement to the “Validity as Social Imperative” framework. Both frameworks may be held together, and evidence of both conceptualizations may be collected in parallel. This tandem approach may be of the most value when addressing concerns of fairness, equity, and consequences resulting from test scores. This is especially true in high-stakes situations, where the consequences of test outcomes are severe, and patient safety may be impacted.

Victoria A. Roach: Conceptualization; writing – review and editing; writing – original draft; resources.

No grants or funding supported the production of this manuscript.