Mixed methods systematic review: Using a cultural validity assessment to evaluate prevention programs for Indigenous students

Implementing culturally appropriate school-based prevention programs with Indigenous students that leverage culture as a protective factor has the potential to revitalize and sustain cultural connections that have historically and systematically been destroyed in the United States. However, there is a dearth of literature synthesizing the effectiveness of school-based prevention programs that have been implemented with Indigenous students across contexts. As such, we conducted a mixed method systematic review to (a) evaluate school-based prevention programs with quantitative and/or qualitative data, (b) assess the use of Indigenous research methods, and (c) examine cultural and community validity. Studies were included if they were published between January 2010 and August 2022, reported quantitative and/or qualitative outcomes for a prevention program implemented in a K–12 school with Indigenous students, or examined an intervention that was designed for Indigenous students (even if there were non-Indigenous students in the study). We strategically included qualitative and mixed methods studies to ensure that cultural and community contexts were represented in this study and to contextualize quantitative findings. Our search resulted in the inclusion of 36 manuscripts describing 28 different interventions. There were 11 mixed methods, three qualitative, and 22 quantitative studies. Quality was assessed using the Mixed Methods Appraisal Tool (Hong et al., 2018) and a randomized controlled trials appraisal tool drawn from the Journal Article Reporting Standards (Appelbaum et al., 2018). Building on the work from Kūkea Shultz and Englert (2021), cultural validity was assessed by conceptualizing cultural validity into the two distinct domains of purposeful engagement and intentional privileging. Intervention effectiveness was evaluated and separated into three criteria (i.e., positive, null, and mixed) to determine if effectiveness varied based on intervention or study design. Most of the studies reported positive outcomes and effectiveness did not vary based on study design (i.e., quantitative, qualitative, and mixed methods) or intervention design (i.e., culturally grounded, culturally adapted or mainstream/no cultural adaptations). We discuss implications of systematic review findings as well as the importance of using mixed methods to understand and contextualize intervention effectiveness when conducting research that relates to identity and culture.