Validity in Question: Is Pharmacy Educational Research Meeting the Standards?

Abstract

Objective

We aimed to quantify the rate of validity evidence reporting for educational testing described in pharmacy education literature and determine if this has changed over a 10-year period.

Methods

Articles published between 2019 and 2021 in 5 pharmacy education journals were reviewed to identify studies which reported the use of educational testing for knowledge, skills, and/or abilities. Two investigators independently screened and coded each included article for validity evidence sources reported based on the Standards for Educational and Psychological Testing.

Results

Overall, 1467 articles were screened, wherein 22% (326/1467) reported use of educational testing. Of the articles included, almost one-third (30%) reported 2 or more sources of validity evidence and another 39% reported only 1 source. Validity evidence for content was most frequently reported (54%), while validity evidence for Internal Structure, including reliability, was reported much less often (17%). Alarmingly, many articles (31%) did not report any source of validity evidence for their student learning assessments. Compared to a decade ago, fewer articles reported validity evidence for their student learning assessments.

Conclusion

Despite the critical role of validity evidence in ensuring accurate interpretation of educational testing, its reporting in pharmacy education literature remains inconsistent and has declined over the past decade. Since accurate test score interpretation requires validity evidence, its absence undermines the credibility and impact of research findings. Scholars, journal reviewers, and journal editors should better ensure research reports include validity evidence when inferences are made from learning assessment scores.