A multi-method study evaluating the inference of compartmental model parameters from a generative agent-based model

Abstract

Calibrating process models such as compartmental SIR Models to real data can be performed using either optimization or Bayesian techniques. To accurately assess the performance of these methods, synthetic outbreak data can be used. All information about the data generative process is known for synthetic data, while when using real data there are many unknowns such as under-reporting of cases or real parameter values. We propose using an agent-based model to generate synthetic data. Calibrating to synthetic datasets created using different agent contact structures can provide us with information on how changes in contact structures impact SIR model parameters. We compare results for two calibration methods: Nelder-Mead an optimization technique and HMC, a Bayesian technique. The analysis finds that the two calibration methods perform similar in terms of accuracy when looking at the Mean Absolute Error, Mean Absolute Scaled Error, and Relative Root Mean Squared Error. Looking at the model parameters, HMC is better able to capture the ground truth parameters then Nelder-Mead. The results of the calibration additionally show that the effective infectious period is sensitive to the changes in contact patterns and the proportion of susceptible individuals in the population. For choosing a calibration method, if overall accuracy is the desired outcome, either method should perform equally well, however, if the aim is to understand and analyse the model parameters HMC is a better choice. Understanding how the effective parameters such as the infectious period changes as contact patterns and vaccination rates change can provide valuable information in understanding how to interpret parameters calibrated from real world data that captures both isolation and vaccination.