Applying the major system change framework to evaluate implementation of rapid healthcare system change: a case study of COVID-19 remote home monitoring services.

Abstract

Background: A framework to evaluate implementation of Major System Change (MSC) in healthcare has been developed and applied to implementation of longer-term system changes. This was the first study to apply the five domains of the MSC framework to rapid healthcare system change. We aimed to: i) evaluate implementation of rapid MSC, using England COVID-19 remote home monitoring services as a case study and ii) consider whether and how the MSC framework can be applied to rapid MSC.

Methods: A mixed-methods rapid evaluation in England, across 28 primary and secondary healthcare sites (October 2020-November 2021; data collection: 4 months). We conducted 126 interviews (5 national leads, 59 staff, 62 patients/carers) and surveyed staff (n = 292) and patients/carers (n = 1069). Service providers completed cost surveys. Aggregated and patient-level national datasets were used to explore enrolment, service use and clinical outcomes. The MSC framework was applied retrospectively. Qualitative data were analysed thematically to explore key themes within each MSC framework domain. Descriptive statistics and multivariate analyses were used to analyse experience, costs, service use and clinical outcomes.

Results: Decision to change/Decision on model: Service development happened concurrently: i) early local development motivated by urgent clinical need, ii) national rollout using standard operating procedures, and iii) local implementation and adaptation. Implementation approach: Services were tailored to local needs to consider patient, staff, organisational and resource factors. Implementation outcomes: Patient enrolment was low (59% services <10%). Service models and implementation approaches varied substantially. Intervention outcomes: No associations found between services and clinical outcomes. Patient and staff experiences were generally positive. However, barriers to delivery and engagement were found; with some groups finding it harder to engage.

Conclusions: Low enrolment rates and substantial variation due to tailoring services to local contexts meant it was not possible to conclusively determine service effectiveness. Process outcomes indicated areas of improvement. The MSC framework can be used to analyse rapid MSC. Implementation and factors influencing implementation may differ to non-rapid contexts (e.g. less uniformity, more tailoring). Our mixed-methods approach could inform future evaluations of large-scale rapid and non-rapid MSC in a range of conditions and services internationally.