Design and evaluation of an interactive quality dashboard for national clinical audit data: a realist evaluation

R. Randell, Natasha Alvarado, Mai Elshehaly, Lynn McVey, R. West, P. Doherty, D. Dowding, A. Farrin, R. Feltbower, C. Gale, J. Greenhalgh, Julia Lake, M. Mamas, R. Walwyn, R. Ruddle
{"title":"Design and evaluation of an interactive quality dashboard for national clinical audit data: a realist evaluation","authors":"R. Randell, Natasha Alvarado, Mai Elshehaly, Lynn McVey, R. West, P. Doherty, D. Dowding, A. Farrin, R. Feltbower, C. Gale, J. Greenhalgh, Julia Lake, M. Mamas, R. Walwyn, R. Ruddle","doi":"10.3310/wbkw4927","DOIUrl":null,"url":null,"abstract":"\n \n National audits aim to reduce variations in quality by stimulating quality improvement. However, varying provider engagement with audit data means that this is not being realised.\n \n \n \n The aim of the study was to develop and evaluate a quality dashboard (i.e. QualDash) to support clinical teams’ and managers’ use of national audit data.\n \n \n \n The study was a realist evaluation and biography of artefacts study.\n \n \n \n The study involved five NHS acute trusts.\n \n \n \n In phase 1, we developed a theory of national audits through interviews. Data use was supported by data access, audit staff skilled to produce data visualisations, data timeliness and quality, and the importance of perceived metrics. Data were mainly used by clinical teams. Organisational-level staff questioned the legitimacy of national audits. In phase 2, QualDash was co-designed and the QualDash theory was developed. QualDash provides interactive customisable visualisations to enable the exploration of relationships between variables. Locating QualDash on site servers gave users control of data upload frequency. In phase 3, we developed an adoption strategy through focus groups. ‘Champions’, awareness-raising through e-bulletins and demonstrations, and quick reference tools were agreed. In phase 4, we tested the QualDash theory using a mixed-methods evaluation. Constraints on use were metric configurations that did not match users’ expectations, affecting champions’ willingness to promote QualDash, and limited computing resources. Easy customisability supported use. The greatest use was where data use was previously constrained. In these contexts, report preparation time was reduced and efforts to improve data quality were supported, although the interrupted time series analysis did not show improved data quality. Twenty-three questionnaires were returned, revealing positive perceptions of ease of use and usefulness. In phase 5, the feasibility of conducting a cluster randomised controlled trial of QualDash was assessed. Interviews were undertaken to understand how QualDash could be revised to support a region-wide Gold Command. Requirements included multiple real-time data sources and functionality to help to identify priorities.\n \n \n \n Audits seeking to widen engagement may find the following strategies beneficial: involving a range of professional groups in choosing metrics; real-time reporting; presenting ‘headline’ metrics important to organisational-level staff; using routinely collected clinical data to populate data fields; and dashboards that help staff to explore and report audit data. Those designing dashboards may find it beneficial to include the following: ‘at a glance’ visualisation of key metrics; visualisations configured in line with existing visualisations that teams use, with clear labelling; functionality that supports the creation of reports and presentations; the ability to explore relationships between variables and drill down to look at subgroups; and low requirements for computing resources. Organisations introducing a dashboard may find the following strategies beneficial: clinical champion to promote use; testing with real data by audit staff; establishing routines for integrating use into work practices; involving audit staff in adoption activities; and allowing customisation.\n \n \n \n The COVID-19 pandemic stopped phase 4 data collection, limiting our ability to further test and refine the QualDash theory. Questionnaire results should be treated with caution because of the small, possibly biased, sample. Control sites for the interrupted time series analysis were not possible because of research and development delays. One intervention site did not submit data. Limited uptake meant that assessing the impact on more measures was not appropriate.\n \n \n \n The extent to which national audit dashboards are used and the strategies national audits use to encourage uptake, a realist review of the impact of dashboards, and rigorous evaluations of the impact of dashboards and the effectiveness of adoption strategies should be explored.\n \n \n \n This study is registered as ISRCTN18289782.\n \n \n \n This project was funded by the National Institute for Health and Care Research (NIHR) Health and Social Care Delivery Research programme and will be published in full in Health and Social Care Delivery Research; Vol. 10, No. 12. See the NIHR Journals Library website for further project information.\n","PeriodicalId":73204,"journal":{"name":"Health and social care delivery research","volume":" ","pages":""},"PeriodicalIF":0.0000,"publicationDate":"2022-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"2","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Health and social care delivery research","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.3310/wbkw4927","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 2

Abstract

National audits aim to reduce variations in quality by stimulating quality improvement. However, varying provider engagement with audit data means that this is not being realised. The aim of the study was to develop and evaluate a quality dashboard (i.e. QualDash) to support clinical teams’ and managers’ use of national audit data. The study was a realist evaluation and biography of artefacts study. The study involved five NHS acute trusts. In phase 1, we developed a theory of national audits through interviews. Data use was supported by data access, audit staff skilled to produce data visualisations, data timeliness and quality, and the importance of perceived metrics. Data were mainly used by clinical teams. Organisational-level staff questioned the legitimacy of national audits. In phase 2, QualDash was co-designed and the QualDash theory was developed. QualDash provides interactive customisable visualisations to enable the exploration of relationships between variables. Locating QualDash on site servers gave users control of data upload frequency. In phase 3, we developed an adoption strategy through focus groups. ‘Champions’, awareness-raising through e-bulletins and demonstrations, and quick reference tools were agreed. In phase 4, we tested the QualDash theory using a mixed-methods evaluation. Constraints on use were metric configurations that did not match users’ expectations, affecting champions’ willingness to promote QualDash, and limited computing resources. Easy customisability supported use. The greatest use was where data use was previously constrained. In these contexts, report preparation time was reduced and efforts to improve data quality were supported, although the interrupted time series analysis did not show improved data quality. Twenty-three questionnaires were returned, revealing positive perceptions of ease of use and usefulness. In phase 5, the feasibility of conducting a cluster randomised controlled trial of QualDash was assessed. Interviews were undertaken to understand how QualDash could be revised to support a region-wide Gold Command. Requirements included multiple real-time data sources and functionality to help to identify priorities. Audits seeking to widen engagement may find the following strategies beneficial: involving a range of professional groups in choosing metrics; real-time reporting; presenting ‘headline’ metrics important to organisational-level staff; using routinely collected clinical data to populate data fields; and dashboards that help staff to explore and report audit data. Those designing dashboards may find it beneficial to include the following: ‘at a glance’ visualisation of key metrics; visualisations configured in line with existing visualisations that teams use, with clear labelling; functionality that supports the creation of reports and presentations; the ability to explore relationships between variables and drill down to look at subgroups; and low requirements for computing resources. Organisations introducing a dashboard may find the following strategies beneficial: clinical champion to promote use; testing with real data by audit staff; establishing routines for integrating use into work practices; involving audit staff in adoption activities; and allowing customisation. The COVID-19 pandemic stopped phase 4 data collection, limiting our ability to further test and refine the QualDash theory. Questionnaire results should be treated with caution because of the small, possibly biased, sample. Control sites for the interrupted time series analysis were not possible because of research and development delays. One intervention site did not submit data. Limited uptake meant that assessing the impact on more measures was not appropriate. The extent to which national audit dashboards are used and the strategies national audits use to encourage uptake, a realist review of the impact of dashboards, and rigorous evaluations of the impact of dashboards and the effectiveness of adoption strategies should be explored. This study is registered as ISRCTN18289782. This project was funded by the National Institute for Health and Care Research (NIHR) Health and Social Care Delivery Research programme and will be published in full in Health and Social Care Delivery Research; Vol. 10, No. 12. See the NIHR Journals Library website for further project information.
国家临床审计数据交互式质量仪表盘的设计与评估:一种现实主义评估
国家审计旨在通过促进质量改进来减少质量差异。然而,不同的提供商参与审计数据意味着这并没有实现。该研究的目的是开发和评估质量仪表板(即QualDash),以支持临床团队和管理人员使用国家审计数据。本研究是一项现实主义的器物评价与传记性研究。这项研究涉及5家NHS急性信托机构。在第一阶段,我们通过访谈发展了国家审计理论。数据使用得到数据访问、审计人员熟练生成数据可视化、数据及时性和质量以及感知指标重要性的支持。数据主要由临床团队使用。组织层面的工作人员质疑国家审计的合法性。在第二阶段,我们共同设计了QualDash,并开发了QualDash理论。QualDash提供了交互式的可定制的可视化,以便探索变量之间的关系。在现场服务器上定位QualDash可以让用户控制数据上传频率。在第三阶段,我们通过焦点小组制定了采用策略。会议商定了“冠军”、通过电子公告和示范提高认识以及快速参考工具。在第4阶段,我们使用混合方法评估了QualDash理论。限制使用的指标配置不符合用户的期望,影响冠军推广QualDash的意愿,以及有限的计算资源。易于定制支持使用。最大的用途是以前数据使用受限的地方。在这些情况下,虽然中断的时间序列分析并未显示数据质量得到改善,但报告编写时间减少了,并支持了提高数据质量的努力。23份问卷被退回,显示了对易用性和有用性的积极看法。在第5阶段,对QualDash进行随机对照试验的可行性进行了评估。我们进行了采访,以了解如何修改QualDash以支持区域范围的黄金司令部。需求包括多个实时数据源和功能,以帮助确定优先级。寻求扩大审计业务的审计可能会发现以下策略是有益的:让一系列专业团体参与选择指标;实时报告;呈现对组织级别员工重要的“标题”指标;使用常规收集的临床数据填充数据字段;以及帮助员工探索和报告审计数据的仪表板。那些设计仪表板的人可能会发现包含以下内容是有益的:关键参数的“一目了然”可视化;可视化配置与团队使用的现有可视化一致,并带有清晰的标签;支持创建报告和演示文稿的功能;探索变量之间的关系并深入查看子组的能力;对计算资源的要求低。引入仪表板的组织可能会发现以下策略有益:临床倡导推广使用;审核人员用真实数据进行测试;建立将使用纳入工作实践的例行程序;让审计人员参与收养活动;并允许定制。COVID-19大流行停止了4级数据收集,限制了我们进一步测试和完善QualDash理论的能力。问卷调查的结果应谨慎对待,因为样本小,可能有偏倚。由于研究和开发的延迟,无法进行中断时间序列分析的对照地点。一个干预站点没有提交数据。有限的吸收意味着评估对更多措施的影响是不适当的。应探讨国家审计指示板的使用程度和国家审计为鼓励采用指示板而使用的战略,对指示板的影响进行现实审查,并对指示板的影响和采用战略的有效性进行严格评估。本研究注册号为ISRCTN18289782。该项目由国家卫生和保健研究所(NIHR)卫生和社会保健提供研究方案资助,将全文发表在《卫生和社会保健提供研究》上;第10卷,第12期请参阅NIHR期刊图书馆网站了解更多项目信息。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
CiteScore
1.00
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信