Explaining the impact of parameter combinations in agent-based models

Abstract

Simulation is a useful and effective way to analyze and study complex, real-world systems, allowing researchers, practitioners, and decision makers to make sense of the inner working of a system involving many factors, often resulting in some sort of emergent behavior. The number of parameter value combinations grows exponentially and it quickly becomes infeasible to test them all or even to explore a suitable subset. How does one then efficiently identify the parameter value combinations that matter for a particular simulation study, and determine their impact on the result? In addition, is it possible to train a machine learning model to predict the outcome of an agent-based model (ABM) with a systematically chosen small subset of parameter value combinations, such that the result could be predicted without running the ABM? We use covering arrays to create $t$-way ($t$ = 2, 3, etc.) combinations of parameter values to significantly reduce an ABM’s parameter value exploration space, which is supported by our prior work. In our ICCS 2023 paper (Olsen et al., 2023) we built on that work by applying it to Wilensky’s Heatbugs model and training a random forest machine learning model to predict simulation results by using the covering arrays to select our training and test data. Our results show that a 2-way covering array provides sufficient training data to train our random forest to predict three different simulation outcomes. Our process of using covering arrays to decrease parameter space to then predict ABM results using machine learning is successful. In this paper that extends the ICCS 2023 paper (Olsen et al., 2023), we analyze the role of parameter combinations and parameter values in determining model output via combination frequency difference (CFD) analysis and Shapley values. CFD has not previously been applied to agent-based models; we provide a process for using this approach and compare and contrast with Shapley values and random forest feature importance.