Validity theory and validation of demographic self-report: a critical quantitative appraisal

Abstract

Advances in different approaches to critical quantitative statistics have been extensively used across various fields, with a wealth of empirical research illustrating applications and results. In education and other disciplines interested in disrupting White supremacy embodied across much of today’s policy and procedures, there is an increasing call to incorporate a diverse range of critical approaches to research methods. This pivoting aims to challenge the marginalization of subaltern ways of knowing, which has resulted in a tradition of reliance on (post-)positivist frameworks that limit the scope of research inference and validity. Critical psychometrics has become central in shaping our understanding of measurement, its accuracy, and the rationale behind approaches to code self-reported demographic categories as stable, fixed, and accurate estimates of oppression. This paper provides two mixed-methods survey study examples where these assumptions unsettle the notion that self-report demographics measure comparative indices of identity. Implications of this critical quantitative psychometric appraisal challenges the taken-for-granted nature of self-reported demographics; that is, these essentialized identity categories should be used to accurately and precisely compare differences between disability, race, class, gender, socioeconomic status, and educational background. With the validity theory behind this assumption challenged, this work provides a starting point, a stake in the ground, to pursue diverse methods to validate self-reports.