Validity for whom? Unpacking assessment validity from an inclusion perspective

Abstract

Weiner et al., in their recent paper ‘The myth that slow test-takers are worse students: Implications for time-limited testing”’,¹ interrogate the myth that those who can complete time-limited assessments rapidly are better students or clinicians. They point to the lack of evidence supporting this myth and the likelihood that time pressure selects for a set of attributes not necessarily aligned with professional competence.

The myth paper illustrates one way that some groups of learners (those who may find time-limited examinations more challenging) encounter barriers within assessment systems despite possessing the underlying capability being assessed. This is a critical example of the validity issues woven into assessment systems, highlighting two questions we should be asking:

The first question points directly to issues of construct validity, the ability of an assessment to evaluate the intended capability or learning outcome (the construct).² However, when observation of construct-relevant factors is obscured by extraneous (construct-irrelevant) factors, the validity and therefore defensibility of assessments fall. Often assessments are designed with a clear outcome in mind, with less consideration of construct-irrelevant factors which might get in the way.

The experiences of learners who diverge from dominant norms of learning, communication and performance provide an important lens through which to explore construct-irrelevant factors which compromise assessment validity. Weiner et al. point to learners who possess the requisite knowledge and skills but perform poorly on time-limited assessments. This might include learners who use English as an additional language or dialect, those with slower language processing, those whose voices do not conform to masculine standards,³ those who find prolonged eye contact distracting or those with limited manual writing/typing endurance. If otherwise capable students perform differently because of factors unrelated to the intended construct, then assessment outcomes may reflect conformity to norms of performance rather than competence itself.

This is fundamentally a construct validity problem. Examining the performance of learners who differ from the ‘ideal student’ reveals what assessments are unintentionally measuring. Neurodivergent students and other minoritised learners report feeling constrained by norms of professionalism, forced to mask, conform and perform.⁴ As Weiner et al. point out that time-limited testing presumes that rapid cognitive processing is valued in healthcare professionals, do our assessments also presume that standardised forms of social fluency and tolerance to sensory stimuli are normal, necessary and valued?

Features such as strict timing, particular communication formats or performance under observation are often defended on the grounds that they reflect the realities of professional practice. In such circumstances, ‘authenticity’ can become a rationale for preserving existing assessment conditions even when questions are raised regarding their relevance to the intended construct. Rather than asking whether an adjustment compromises authenticity, a validity-oriented approach asks whether the aspect of assessment in question is essential to the capability being assessed.

This brings us to the second question posed above. When we critically consider what assessments measure, we are also invited to consider what those assessments produce: their consequential validity. Consequential validity refers to the impacts (beyond the immediate act of assessment) on students and learning.⁵ This calls us to consider the effects of our measurement practices. For example, when we design assessments prioritising speed, we may reproduce worlds of work which constrain health professionals to act with constant urgency, rapid task switching and unilateral decision making. These assessments may unintentionally equate speed with competence and perpetuate overwork culture seen in understaffed services.⁶

Labelling an assessment as authentic does not merely hold up a mirror to professional practice; it shapes that practice and who is seen as legitimate, reinforcing norms and constraining identity and behaviour.⁷ As Marceau et al. note, ‘assessment is a social act and has social consequences’.⁸ McArthur (2022) suggests that authenticity in assessment can reflect what we want to see in the world of work, not simply what it currently is.⁹ If assessments contribute to shaping professional identities, workplace norms and understandings of competence, then consequential validity invites us to think not only about what assessment reproduces but also about what it makes possible. Rather than viewing authenticity as replicating current professional realities, health professions educators might ask how assessment can support development of more equitable, inclusive and sustainable forms of practice that better understand the lived experiences of diverse patient populations.

Inclusive, authentic assessment design then becomes less concerned with adapting learners to existing assessment structures and more concerned with critically examining assessment conditions and expectations to determine what is genuinely required for safe and effective practice. Such a shift requires attention to both construct and consequential validity: clarifying what capabilities and knowledge are important to assess, which capabilities are truly being assessed, and considering how assessment design influences whose competence becomes visible, valued and recognised within the profession.¹⁰

A validity-oriented approach to those considerations might involve first reviewing assessment and program learning outcomes to ensure they reflect those core capabilities. Then, considering what assessment formats provide the most direct, valid evidence of those outcomes. Finally, exploring how construct-irrelevant assessment features can be removed, reduced or accommodated. Where construct-irrelevant features are retained due to practical constraints such as time, staffing or resources, this should be acknowledged explicitly. Transparently distinguishing resource limitations from essential features of professional practice aims to inhibit evolving mythologies, creating opportunities for negotiation, improvement and more equitable assessment practice over time.

This process does not remove appropriate assessment challenge or complexity. Rather, it requires critically examining potentially deeply embedded assumptions.

Particularly relevant to clinical assessment are communication assumptions. Communication is not universal across cultures, populations or indeed neurotypes.¹¹ Rather than rewarding normative social behaviours or particular communication styles, assessments could focus on clarity, safety, responsiveness and effectiveness. This may involve flexibility in how competence is demonstrated, for example, through verbal handover, written documentation or other professionally relevant communication modes.

Underpinning many assumptions is a deeper cultural understanding which positions the ‘ideal student’ at the centre of assessment design.¹² In health professions education, the qualities rewarded through assessment help construct an image of the ‘ideal professional’, making assessment a powerful site of professional identity formation. Examining who is readily recognised within these constructions, and who is at their margins, offers an important lens through which to rethink assessment and the resulting professional identities.

Attention to these factors might allow us to build systems of assessment which improve construct validity, more accurately evaluating the performance of all learners. Perhaps equally important, reshaping assessment practices for inclusion might enable us to create worlds of work and learning that acknowledge and build greater breadth and diversity in our future health workforce.

Laura Gray: Conceptualization; writing—original draft; writing—review and editing. Sue A. Sharpe: Conceptualization; writing—original draft; writing—review and editing.

Data sharing is not applicable to this article as no datasets were generated or analysed during the current study.