Preliminary evaluation of a questionnaire for assessing fidelity of early intervention for psychosis services.

Purpose: Fast, easy, and cost-effective methods are needed for fidelity assessment, quality improvement initiatives, and population-based studies in Early Intervention for Psychosis (EIP) services. Having an online questionnaire assessing the fidelity of EIP services, completed by staff self-reports, and having evidence of reliability and validity, could fill that gap. We assess the reliability and validity of the Early Intervention for Psychosis Services Fidelity Questionnaire (EIPS-FQ), developed in Part I of this set of papers.

Methods: A convenience sample of 10 EIP teams in England was used. Two staff members completed online questionnaires assessing recent and past fidelity. An external rater completed the same questionnaire for the two time periods, using a random sample of patient medical records, program documentation, and interviews with staff. The intra-class correlation coefficient (ICC) was calculated to assess inter-rater reliability. Validity was assessed using Bland-Altman plots, absolute mean differences, and the ICC.

Results: The fidelity score measuring recent fidelity ranged from 54.2 to 82.7, out of a possible 100. The ICC assessing reliability of the fidelity score was 0.40 (95% CI: 0.0-0.81). The ICCs for the fidelity sub-category scores ranged from 0 to 0.76. Two sub-categories, comprehensive assessments and family involvement and intervention, had low ICCs, regardless of period examined.

Conclusions: This first attempt at validating the EIPS-FQ has demonstrated that the reliability of the EIPS-FQ is moderate/low, and therefore requires modification prior to use. The next iteration of the fidelity questionnaire will clarify or remove items which had very low reliability and add evidence-based components not identified in the Delphi exercise.