Evaluating the effectiveness of vaccination campaigns: Insights from unvaccinated mortality data

Abstract

This paper examines a recently developed statistical approach for evaluating the effectiveness of vaccination campaigns in terms of deaths averted. The statistical approach makes predictions by comparing death rates in the vaccinated and unvaccinated populations. The statistical approach is preferred for its simplicity and straightforwardness, especially when compared to the difficulties involved when fitting the many parameters of a dynamic SIRD-type model, which may even be an impossible task.

We compared the estimated number of deaths averted by the statistical approach to the “ground truth” number of deaths averted in a relatively simple scheme (e.g., constant vaccination, constant $R_0$, pure SIR dynamics, no age stratification) through mathematical analysis, and quantified the difference and degree of underestimation. The results indicate that the statistical approach consistently produces conservative estimates and will always underestimate the number of deaths averted by the direct effect of vaccination, and thus obviously the combined total effect (direct and indirect effect).

For high $R_0$ values (e.g. $R_0\ge$ 8), the underestimation is relatively small as long as the vaccination level ($v$) remains below the herd immunity vaccination threshold. However, for low $R_0$ values (e.g. $R_0\le$ 1.5), the statistical approach significantly underestimates the number of deaths averted by vaccination, with the underestimation greater than 20%. Applying an approximate correction to the statistical approach, however, can improve the accuracy of estimates for low $R_0$ and low $v$.

In conclusion, the statistical approach can provide reasonable estimates in scenarios involving high $R_0$ values and low $v$, such as during the Omicron variant epidemic in Australia. For low $R_0$ values and low $v$, applying an approximate correction to the statistical approach can lead to more accurate estimates, although there are caveats even for this. These results suggest that the statistical method needs to be used with caution.