Strengthening research preparedness for crises: lessons from Norwegian government agencies in using randomized trials and quasi-experimental methods to evaluate public policy interventions.

Abstract

During public health crises such as pandemics, governments must rapidly adopt and implement wide-reaching policies and programs ("public policy interventions"). A key takeaway from the coronavirus disease 2019 (COVID-19) pandemic was that although numerous randomized controlled trials (RCTs) focussed on drugs and vaccines, few policy experiments were conducted to evaluate effects of public policy interventions across various sectors on viral transmission and other consequences. Moreover, many quasi-experimental studies were of spurious quality, thus proving unhelpful for informing public policy. The pandemic highlighted the need to continuously develop competence, capacity and a robust legal-ethical foundation for impact evaluations well before crises occur. It raised a crucial question: how can governments in non-crisis times develop capabilities to generate evidence on the effects of public policy interventions, thus enabling a rapid and effective research response during public health crises? We conducted a mapping to explore how government agencies in Norway use RCTs and quasi-experimental methods to strengthen the evidence base for public policy interventions and to identify barriers and facilitators to their use. Contributing to the study were 10 government agencies across sectors such as development assistance, education, health, social welfare, statistics and taxation. Many of these agencies have conducted or commissioned RCTs or quasi-experimental studies in the past 5 years, with evaluations ranging from 1 or 2 to more than 15 per agency. The measures evaluated included organizational, educational and financial interventions and interventions for oversight and sanctions. Some agencies have internal capabilities for designing and conducting evaluations, while others commissioned such studies to universities and other research institutions. Agencies reported examples where enhanced communication among implementers, researchers, ministries and political leaders facilitated impact evaluations, and these lessons offer opportunities for cross-sector knowledge-sharing to help strengthen rigorous evaluations of public policy interventions. Despite their potential, various government agencies report that randomized and quasi-experimental studies face legal, ethical, political and practical barriers that affect their use. For instance, the urgency of politicians to implement policies at scale has led to the discontinuation of trials and hindered learning from their effects. The surveyed agencies stressed the importance of legislation providing clear guidelines on when differential treatment can be justified and when informed consent requirements can be waived, as well as faster and clearer processes for managing privacy concerns related to data access. Crucially, greater political acceptance for systematically and gradually implementing reforms, including using randomization, could strengthen evidence-informed public policy, enhancing the scaling-up of effective interventions and deprioritizing ineffective ones.