The psychometric properties and treatment outcomes associated with two measures of the adult therapeutic alliance using naturalistic data

Purpose

This is the first of two articles that examine the psychometric properties and treatment outcomes associated with measures of the therapeutic alliance collected in naturalistic settings rather than clinical trials. These articles break new ground in understanding the relationship between measures of alliance and treatment outcome by performing detailed item analyses in order to assess the psychometric properties of each individual item, its propensity to change from session to session and its utility in predicting change in the outcome measure.

Methods

Data were taken from the ACORN database for adults attending for psychotherapy treatment in naturalistic settings (N = 147,399). The sample, the largest to date, included only those completing both an alliance measure and an outcome measure at every session. Two sets of three different alliance items are used across three different treatment populations: general outpatient, substance use, and severe and persistent mental health difficulties. To investigate this possibility, the psychometric properties of each item were evaluated, including factor analysis, likelihood of change over time in treatment and correlations between changes in alliance and magnitude of client-reported improvement in therapy. The predictive validity of each individual item is compared with the predictive validity of their respective three-item measures. The problem of alliance scores being heavily skewed in a positive direction and the resulting lack of variability from session to session is addressed by categorising change in alliance as same, better or worse, regardless of the magnitude of change.

Results

Findings tended to differ from those in clinical trials, with the last alliance score being most predictive of outcome. In this data set, the alliance accounted for, at most, 2% of the variance in outcomes. Measures of the therapeutic alliance demonstrate ceiling effects, and the alliance–outcome correlation is far from linear. Change in alliance score, rather than a single assessment, is more predictive of outcome, regardless of the magnitude of change in the alliance measure, with effect sizes of up to 50% more for those who rated the alliance as worse than for those rating it as no change or better.

Conclusion

Therapists using therapeutic alliance questionnaires will benefit from being aware of how the psychometric properties of alliance measures impact outcomes. Even the smallest drop in alliance is predictive of significant clinically meaningful differences. Implications for practice, training and research are considered.