Assessing multidimensional complex decision making with Situational Judgment Tests

The decisions that humans make are often multidimensional and complex, and system aids have been developed to support decision-making. However it is often difficult to evaluate the decision making of both the human or these system decision aids as decision making is a skill that is difficult to quantify. Traditionally, decision-making skill is assessed primarily through tasks or questionnaires. The challenge with those approaches is that measures of decision-making skill are often unidimensional. The Situation Judgment Test (SJT) is multidimensional and comprises scenarios obtained from accounts of real-world experiences, each with response options from which respondents select the most effective response. Although SJTs often show criterion-validity, this is typically obtained from post-hoc analysis. The present study seeks to evaluate the dimensionality of SJT scenarios, which are based on real-world decisions that are akin to the type of decisions that decision support systems seek to aid with. Fifteen SJT scenarios were administered to 94 participants, along with several measures of dimensions deemed relevant to real-world decision making. Most of the dimensions were able to predict performance on at least one scenario. Certain dimensions seemed to predict performance on more scenarios than others did. The results indicated that the SJT scenarios were able to incorporate some dimensions relevant to decision making. Future research should examine other measures for evaluating SJTs prior to using them to assess decision making.