A Simulation-Based Comparison of Deterministic and Stochastic Multicriteria Models: Analyzing Rank Divergence

Although Stochastic Multicriteria Acceptability Analysis (SMAA) has been widely applied in real-world decision problems, limited research has examined the structural conditions that lead to rank disagreement between deterministic and stochastic model outputs. This paper addresses that gap through a simulation-based analysis of 50 randomly generated decision problems. First, one-hot encoded vectors were developed to compare the deterministic top-ranked alternatives with their SMAA rank acceptability distributions to evaluate rank divergence. Descriptive statistics showed that cases with disagreement had a substantially higher mean Jensen–Shannon Distance (JSD) (0.79) in comparison to non-divergent cases (0.43). Moreover, scatterplot analysis revealed that divergent cases typically have high JSD values (≥ 0.6), low rank-1 acceptability (≤ 0.2), and high rank expectation (≥ 4). Second, statistical techniques were used to compare differences between structural features, i.e., criteria, alternatives, minimum and maximum criteria. Furthermore, the Criteria Balance Score (CBS) was developed to quantify criteria type imbalance, where values of 0 show perfect balance and scores close to 1 demonstrate disparity. Results showed that divergent cases included decision problems with statistically significant larger model complexity, i.e., number of criteria, and criteria type min/max balance, which was an unexpected finding. Third, threshold-based analyses revealed that 62.5% of divergent cases included decision structures with 10 or more criteria, and that 75% of diverging cases with CBS below 0.20 had a min/max criteria type difference of 0 or 1. Finally, consistency in divergence patterns was independently explored within four multicriteria decision analysis models. Findings suggest that divergence is largely a function of decision space characteristics, rather than idiosyncrasies of individual models. Together, these findings provide real-world decision makers, analysts, and researchers with practical, evidence-based thresholds for instances when deterministic results may not be robust. By identifying these structural warnings in advance, decision makers can increase stakeholder trust and reliability in the decision-making process.