Confounding and Negative Control Methods in Observational Study of SARS-CoV-2 Vaccine Effectiveness: A Nationwide, Population-Based Danish Health Registry Study

Background: Observational studies of SARS-CoV-2 vaccine effectiveness are prone to confounding, which can be illustrated using negative control methods.
Methods: Nationwide population-based cohort study including two cohorts of Danish residents 60– 90 years of age matched 1:1 on age and sex: A vaccinated and a non-vaccinated cohort, including 61052 SARS-CoV-2 vaccinated individuals between 1 March and 1 July 2021 and 61052 individuals not vaccinated preceding 1 July 2021. From these two cohorts, we constructed negative control cohorts of individuals diagnosed with SARS-CoV-2 infection or acute myocardial infarction, stroke, cancer, low energy fracture, or head-trauma. Outcomes were SARS-CoV-2 infection, negative control outcomes (eg, mammography, prostate biopsy, operation for cataract, malignant melanoma, examination of eye and ear), and death. We used Cox regression to calculate adjusted incidence and mortality rate ratios (aIRR and aMRR).
Results: Risks of SARS-CoV2 infection and all negative control outcomes were elevated in the vaccinated population, ranging from an aIRR of 1.15 (95% CI: 1.09– 1.21) for eye examinations to 3.05 (95% CI: 2.24– 4.14) for malignant melanoma. Conversely, the risk of death in the SARS-CoV-2 infected cohort and in all negative control cohorts was lower in vaccinated individuals, ranging from an aMRR of 0.23 (95% CI: 0.19– 0.26) after SARS-CoV-2 infection to 0.50 (95% CI: 0.37– 0.67) after stroke.
Conclusion: Our findings indicate that observational studies of SARS-CoV-2 vaccine effectiveness may be subject to substantial confounding. Therefore, randomized trials are essential to establish vaccine efficacy after the emergence of new SARS-CoV-2 variants and the rollout of multiple booster vaccines.

Plain Language Summary: Why was this study done: After the emergence of new SARS-CoV-2 variants and the rollout of multiple booster SARS-CoV-2 vaccines, the impact of vaccination on risk of SARS-CoV-2 infection and death after the infection has mainly been explored in observational studies. We used negative control methods to investigate whether confounding affects the results of observational SARS-CoV-2 vaccine effectiveness studies.
Findings: We used Danish registry data obtained during the SARS-CoV-2 vaccine roll-out to conduct a nationwide, matched population-based cohort study of Danish residents 60– 90 years in which we compared vaccinated individuals with non-vaccinated individuals. Compared with unvaccinated individuals, vaccinated individuals had increased risks of SARS-CoV2 infection but also had increased risks of all negative control outcomes (mammography, prostate biopsy, operation for cataract, malignant melanoma, examination of eye and ear). The risk of death after SARS-CoV2 infection was lower in the vaccinated cohort, as was the risk of death after acute myocardial infarction, stroke, cancer, low energy fracture, and head-trauma.
Meaning: The negative control methods indicate that observational studies of SARS-CoV-2 vaccine effectiveness may be prone to substantial confounding which may impact the observed associations. This bias may both lead to underestimation of vaccine effectiveness (increased risk of SARS-CoV2 infection among vaccinated individuals) and overestimation of the vaccine effectiveness (decreased risk of death after of SARS-CoV2 infection among vaccinated individuals). Our results highlight the need for randomized vaccine efficacy studies after the emergence of new SARS-CoV-2 variants and the rollout of multiple booster vaccines.