A comparison of scalable routine clinical materials and observer ratings to assess CBT fidelity

Abstract

Decades of research have demonstrated the efficacy of cognitive behavioral therapies (CBTs) for a wide variety of psychiatric diagnoses, resulting in the inclusion of CBT as a first-line evidence-based practice (EBP) in treatment guidelines for mood and anxiety disorders. However, some research suggests that many providers do not implement EBPs as intended. Ongoing quality monitoring is needed to support EBP implementation and sustainability, but “gold standard” fidelity monitoring (e.g. observer ratings) is time-consuming, requires extensive training, and may feel intrusive to providers and clients. In the current study, we aimed to develop a scalable method of assessing CBT fidelity that leverages information generated in routine clinical care (e.g. session worksheets and clinician checklists). Ratings of adherence based on worksheets were not correlated with ratings of adherence based on observer ratings. However, ratings of competence based on worksheets were significantly correlated with observer ratings of competence. Ratings of adherence based on clinician checklist ratings were also significantly correlated with observer-rated adherence. Results did not indicate a strong relationship between adherence or competence measured by worksheet ratings or observer and symptom change. However, adherence as measured by clinician checklists were associated with subsequent depression symptom change. Findings have a strong potential to impact fidelity monitoring strategies for a variety of CBTs. Given the limited resources and time to do full audio review in routine care settings, findings suggest that using routine materials generated in session to assess therapist competence may be a feasible alternative to the “gold standard” audio review. The trial is registered at ClinicalTrials.gov, number NCT03479398.