Reliable crash analysis: Comparing biases and error rates of empirical Bayes before-after analyses to mixed-models

Abstract

Estimating reliable causal estimates of road safety interventions is challenging, with a number of these challenges addressable through analysis choices. At a minimum, developing reliable crash modification factors (CMFs) needs to address three critical confounding factors, i.e., 1) the regression-to-the-mean (RTM) phenomenon, 2) the effect of traffic volume, and 3) the time trend for the occurrence of crashes. The current preferred crash analysis method is the empirical Bayes (EB) before-after analysis but requires complex bespoke analysis and may not be the best performing method. We compare in a simulation experiment various EB methods to a more straightforward negative binomial generalized linear mixed model (NB-GLMM) with an interaction term between treatment group and time for analysing treatment effects in crash data. Data were simulated using two broad scenarios: 1) an idealized randomized controlled design, and 2) a moderately biased site-selection scenario commonly encountered in road safety crash analyses. Both scenarios varied treatment effects, overdispersion, and sample sizes. The NB-GLMM performed best, maintaining type I error rate and providing least biased estimates across most analyses. Most standard EB methods were too liberal or generally more biased, with the exception of the EB method that incorporated a varying dispersion parameter. Incorporating mixed-effects modelling into the EB procedure improved bias. Overall, we found that using a “standard” NB-GLMM with an interaction term is sufficient for crash analysis, reducing complexity compared to bespoke EB solutions. Chosen methods should also be the least biased and possess the marginal error rates under both ideal and selection-bias conditions. Mixed-effects approaches to analysis of road safety interventions satisfy these criteria outperforming standard or other empirical Bayes approaches tested here.