The use of a matched controls design and publicly available outcome data to evaluate a multiyear urban school-based SEL intervention

Abstract

This study evaluated the effects of the Mastering Our Skills and Inspiring Character (MOSAIC) intervention on school-wide student outcomes for urban middle school students. Specifically, it investigated the extent to which schools implementing the MOSAIC intervention (n = 6) show greater progress on key state-collected academic (e.g., English Language Arts, Mathematics) and behavioral (e.g., absenteeism, behavior referrals) outcome variables, compared to matched schools not hosting MOSAIC (n = 30). Data were collected from publicly available school performance reports across multiple time points, including baseline, early implementation, and full implementation. These indicators were assessed annually, with academic assessments administered in the spring and behavioral metrics aggregated for the full academic year. This study utilizes the overlapping matched control group design, an innovative evaluation approach that involves fewer schools than most randomized control trials and relies on measurements routinely collected by schools, so as to provide feasible adoption of this approach by local schools and districts. Additionally, this study adds to the emerging literature on the impact of longitudinal SEL-related interventions on behavioral and academic trajectories in the middle grades with predominantly urban, minoritized youth, to inform educational policy. Our findings indicate mixed results across academic and behavioral indicators, such as significant improvement in Math scores but smaller effects for behavioral metrics, such as substance use. The results highlight the challenges and potential value of the matched control approach and the importance of monitoring multiple domains to fully understand intervention impacts.