Psychometric Evaluation of a Pragmatic Measure for Assessing Adherence to System of Care Principles in Behavioral Health Service Interactions

Abstract

Provider adherence to system of care principles in service interactions with families is an important indicator of behavioral health service quality for youth; however, valid and pragmatic measures suitable for monitoring this quality indicator at population scale have not been developed. This article reports on two studies that developed and evaluated such a measure. In Study 1, an iterative, family-partnered process resulted in generation of 18 items that demonstrated unidimensionality and strong reliability among caregivers of youth participating in behavioral health services (N = 141). In Study 2, data from a second, statewide, stratified random sample of caregivers (N = 351) confirmed the items’ unidimensionality, discriminant validity, and criterion-related validity. Higher scores on the System of Care Adherence Scale were associated with lower risk of youth psychiatric hospitalization, greater perceived improvement in youth functioning, and greater increases in caregivers’ self-efficacy to access services. Item response theory analyses indicated the items were strongly related to adherence; however, most were optimal for differentiating between low to moderate levels of adherence. The System of Care Adherence Scale is a psychometrically sound measure suitable for population surveillance of the extent to which families experience system of care principles in their interactions with providers.