Should We Use Clinician Self-Report to Tailor Implementation Strategies? Predicting Use of Youth CBT with Clinician Self-Report Versus Direct Observation.

Abstract

Developing tailored implementation strategies to increase the use of evidence-based practice (EBP) requires accurate identification of predictors of their use. However, known difficulties with measuring EBP use complicates interpretation of the extant literature. In this proof-of-concept study, we examined whether the same predictors of use of cognitive behavioral therapy (CBT) are identified when CBT use is measured with clinician self-report compared to direct observation. We examined four candidate predictors of CBT use - clinician participation in an EBP training initiative, years of experience, caseload, and employment status - in a sample of 36 clinicians (64% female; 72% White and 28% Black) from 19 community mental health agencies treating youth in greater Philadelphia. CBT use was captured for 100 unique client sessions (M = 2.8 recorded sessions per clinician) through both clinician self-report and direct observation, using parallel measures. We used three-level (client, clinician, and agency) regression models with random intercepts to estimate the relationship between each predictor variable and CBT use in both measures and compared the magnitude and direction of each model across self-report and direct observation using z-tests. There was no alignment for any of the four candidate predictors between predictive relationships identified by self-report compared to those identified by direct observation. The findings in this study extend literature documenting limitations of using clinician self-report to capture clinician behavior and suggest that even the characteristics that predict higher self-reported CBT use do not align with (and often are discordant with) those that predict directly observed CBT use. This raises questions about the utility of relying on self-reported use to inform implementation strategy design.