The Role of Expertise in Scoring Situational Judgment Tests: Is the Juice Worth the Squeeze?

Drawing from theory regarding the general and domain-specific knowledge typically measured by situational judgment tests (SJTs), this study examined the effects of subject matter experts (SME) qualifications on SJT scoring keys and their associated predictive validity. Although one may expect that scoring keys generated using the judgments of more qualified SMEs would result in higher predictive validity, an exploration of any gains in incremental validity associated with the keys is warranted to determine whether gains are meaningful, and perhaps more importantly, worth the additional resources required to obtain highly qualified SMEs. We created three distinct SJT scoring keys for an SJT designed to measure cross-cultural competence based on a sample of crowdsourced novices, a sample of incumbents, and a sample of highly trained and experienced SMEs and examined their predictive validity. Findings from a time-lagged, year-long concurrent validity study (N = 350) provide some support for the idea that using particularly qualified, highly experienced SMEs to develop SJT scoring keys may provide a meaningful increase in the predictive validity of the assessment over using a crowdsourced novice sample or a convenience sample of incumbents that are often used in practice.