Opening the black box of psychometrics

Abstract

The term “psychometrics” is inherently intimidating to the average health sciences educator and scholar, and yet, the foundational principles of the discipline—determining the impact of latent variables that contribute to human behavior—are important for determining whether the tools we use as educators and educational scholars are consistent at measuring what they were intended to measure. The terms “reliability” and “validity” themselves conjure images of head-scratching and question marks in bubble clouds. In the simplest of terms, reliability is a mathematical term for precision, whereas validity is a mathematical term for accuracy. The dartboard analogy visualization is apropos here: Darts thrown in the same vicinity on a dartboard reflect precision or reliability; darts thrown at the intended target of the bullseye reflect accuracy or validity. In an ideal world, we want our tools measuring performance to be both reliable and valid.

Need to know if a rater measuring performance on an Objective Structured Clinical Exam (OSCE) is observing the same phenomena across students based on a grading rubric?Perform a test–retest analysis. Need to know if two different raters for that same OSCE are observing the same phenomena? Perform an inter-rater reliability analysis. Need to know if that survey you used as a data collection instrument is reliable across subjects? Perform a Cronbach's alpha analysis. Need to know if an exam you wrote was written clearly without introducing unintended bias and measuring what you intended to assess? Perform an item analysis. These are just some examples of why psychometrics matter.

There are so many different analyses that can be performed to measure reliability and validity. This alone is an intimidating attribute of psychometrics. Add the fact that all of these analyses are mathematically based, and that leaves educators and scholars in the black box of analysis trust and an urge to seek out the resident statistician. (But there's hope! The math is all based on “y = mx + b,” which is a generalized linear model, so it is really not that bad!) We did not go into health sciences education for the math, after all, so why all the fuss?!

Why all the fuss is exactly what this month's Guest Editor, Dr. Adam Wilson of Rush University in Chicago, IL, has explored in the pages that follow. One of us (JMO) has known Dr. Wilson since the last days of his graduate work at Indiana University, where even then he was pushing anatomy educators to think more critically about our work. In the years since he graduated, he has become one of the world's leading experts in quantitative approaches to enhancing anatomy, medical, and health education. This is one of the primary reasons he was invited into an Associate Editor role at Anatomical Sciences Education, a role he has used to elevate the quality of scholarship published in our pages. Our journal has a long history of publishing psychometric work,^1-10 including several important articles from Dr. Wilson's research group.^11-19 This month's special issue expands on those studies and brings psychometrics to the forefront so we can all begin to think about enhancing outcomes for our learners with more rigor.

The most important thing to know about psychometrics was stated eloquently by Dr. Wilson in his introduction to this month's issue¹⁹: “No one wants to make a high-stakes decision about a learner only to later discover the instrument upon which the decision was made failed to accurately or reliably portray the individual's knowledge, skills, behaviors, or perceptions.” He continues by saying, “If there is one takeaway worth emphasizing in this special issue, it is that the quality of research conclusions and data-driven decisions is only as good as the foundational psychometric evidence upon which these arguments are built. Such foundational evidence ought never to be assumed and should always be transparently disclosed for proper contextualization.” Wise words for sure. These are principles to which all educators and educational scholars aspire.

If you want to become a better medical sciences educator and scholar, learning what is in the black box of psychometrics is as essential to you as learning how a generalized linear model is used to compare means of sample populations for a basic scientist or clinical scientist. The difference here is that educators and educational scholars are measuring human performance, which can determine the path and success of individual students. In our estimation, as educators and humanists, the impact has significant ramifications.

This special issue was assembled with you in mind, to help you understand better why psychometrics should be important to you and how it informs your impact as an educator and scholar on the lives of your students. So, dive into that black box!

Jonathan J. Wisco: Conceptualization; writing – original draft; writing – review and editing; methodology; resources; supervision. Jason Organ: Conceptualization; writing – original draft; methodology; writing – review and editing; resources; supervision.

N/A.

The authors declare no conflicts of interest.

N/A.

N/A.