External and Internal Validity Considerations in Youth Effectiveness Trials: Lessons Learned from the COMET Study.

Abstract

Objective: Effectiveness trials aim to increase the generalizability and public health impact of interventions. However, challenges associated with this design present threats to external and internal validity. This paper illustrates these challenges using data from a two-site randomized effectiveness trial, the Community Study of Outcome Monitoring for Emotional Disorders in Teens (COMET) and presents recommendations for future research.

Method: COMET was a randomized effectiveness trial conducted in 19 community mental health clinics in two states comparing three interventions: treatment as usual (TAU), TAU with measurement-based care (TAU+), and the Unified Protocol forTransdiagnostic Treatment of Emotional Disorders in Adolescents with MBC (UPA). Participants included 176 clinicians (mean age = 35.5; 85.8% cisgender female; 53.0% racially and/or ethnically minorized) and 196 adolescents (mean age = 14.7; 65.3% cisgender female; 69.4% racially and/or ethnically minorized). Analyses outlined participant flow from recruitment to study completion, described participant characteristics, and examined site differences.

Results: Analysis of participant flow suggested that recruitment and retention of clinicians and adolescents was challenging, raising questions about whether participants were representative of participating clinics. Both the clinician and adolescent samples were racially and ethnically diverse and adolescents were low income and clinically complex. Significant site differences were observed in clinician and adolescent characteristics.

Conclusions: While this study was successful in recruiting a diverse and historically under-represented sample, difficulties in recruitment and retention raise questions about external validity and site differences present challenges to internal validity of study findings. Suggestions for future effectiveness studies, drawing from implementation science approaches, are discussed.