How a strong measurement validity review can go astray: A look at Higgins et al. (2024) and recommendations for future measurement-focused reviews

Abstract

Critical reviews of a test's measurement validity are valuable scientific contributions, yet even strong reviews can be undermined by subtle problems in how evidence is compiled and presented to readers. First, if discussions of poor reporting practices by a test's users are interwoven with discussions about validity support for the test itself, readers can be inadvertently misled into impressions of the latter which are improperly conflated with the former. Second, test reviewers should give at least as much careful attention to a test's external validity as to its structural validity; test reviewers who prioritize factor analysis and internal consistency at the expense of discriminant and convergent validity can inadvertently mislead readers into perceptions of a test which are more negative or more positive than is warranted by the evidence overall. In this commentary, we aim to help test evaluators in crafting critical investigations of measurement validity. We use Higgins et al.'s (2024) review of the Reading the Mind in the Eyes Test (RMET; Baron-Cohen et al., 2001) as a basis for discussion. We argue that their otherwise impressive review went astray in the two ways described above. After considering both the psychometric evidence that Higgins et al. (2024) provided and the external validity evidence that they did not provide, we conclude that their recommendations that the RMET should be abandoned, and that most prior research findings based on it should be reassessed or disregarded, are unwarranted.