A systematic review informing recommendations for assessing implementation variability in universal, school-based social and emotional learning interventions

There is theoretical support for, and emerging empirical evidence that, implementation variability (e.g., fidelity, dosage, quality) influences outcomes of school-based social-emotional learning (SEL) interventions, yet this relationship remains underexplored. This review aimed to (1) identify and appraise the quality of methods used to assess the relationship between implementation variability and student outcomes and (2) determine the association between implementation dimensions and student outcomes, to reduce the research-to-practice gap and advance evidence-based practice.

Methods

British Education Index, ERIC, PsycINFO, ASSIA, ScienceDirect, and Web of Science were searched, initially identifying 2987 studies. An Implementation Quality Appraisal Checklist (IQAC) was developed to assess the quality of research statistically examining the implementation-outcomes relationship. Extracted data were grouped according to the implementation dimension(s) assessed, the outcome domain(s) examined, and the statistical method(s) used.

Results

Thirty-one studies met the review inclusion criteria. Quality assessment classified fourteen studies (45 %) as low quality, fifteen (48 %) as medium quality, and 2 (7 %) as high quality. The most frequently examined implementation dimensions were dosage (n = 16), fidelity (n = 11), quality (n = 11), responsiveness (n = 6) and reach (n = 3). Inferring the implementation-outcomes relationship was hindered by the heterogeneity and low quality of studies, resulting in a small sample size of comparison groups; calculation of meta-aggregative effect sizes was therefore not possible.

Discussion

This review reveals the paucity of high-quality research examining the relationship between implementation variability of SEL interventions and student outcomes. We propose the use of the aforementioned IQAC to support and guide future research in this area, and provide recommendations to advance implementation science.

Impact statements

• •
  This review reveals that the relationship between implementation variability and child outcomes in universal, school-based SEL interventions remains largely inconclusive due to limited attention, and poor-quality approaches, examining this relationship.
• •
  We call for more and high-quality studies examining the relationship between implementation processes and outcomes.
• •
  Alongside this call for action, we offer recommendations to advance implementation process evaluations, actionable insights for researchers and their partners/collaborators, and provide a methodological framework and a quality appraisal checklist to guide and support researchers in conducting a high-quality quantitative assessment of the implementation-outcomes relationship from design through to analysis and reporting.