Systematic review of the performance evaluation of clinicians with or without the aid of machine learning clinical decision support system.

IF 3.1 Q2 MEDICAL INFORMATICS

Health and Technology Pub Date : 2023-06-13 DOI:10.1007/s12553-023-00763-1

Mikko Nuutinen, Riikka-Leena Leskelä

{"title":"Systematic review of the performance evaluation of clinicians with or without the aid of machine learning clinical decision support system.","authors":"Mikko Nuutinen, Riikka-Leena Leskelä","doi":"10.1007/s12553-023-00763-1","DOIUrl":null,"url":null,"abstract":"Background: For the adoption of machine learning clinical decision support systems (ML-CDSS) it is critical to understand the performance aid of the ML-CDSS. However, it is not trivial, how the performance aid should be evaluated. To design reliable performance evaluation study, both the knowledge from the practical framework of experimental study design and the understanding of domain specific design factors are required.Objective: The aim of this review study was to form a practical framework and identify key design factors for experimental design in evaluating the performance of clinicians with or without the aid of ML-CDSS.Methods: The study was based on published ML-CDSS performance evaluation studies. We systematically searched articles published between January 2016 and December 2022. From the articles we collected a set of design factors. Only the articles comparing the performance of clinicians with or without the aid of ML-CDSS using experimental study methods were considered.Results: The identified key design factors for the practical framework of ML-CDSS experimental study design were performance measures, user interface, ground truth data and the selection of samples and participants. In addition, we identified the importance of randomization, crossover design and training and practice rounds. Previous studies had shortcomings in the rationale and documentation of choices regarding the number of participants and the duration of the experiment.Conclusion: The design factors of ML-CDSS experimental study are interdependent and all factors must be considered in individual choices.Supplementary information: The online version contains supplementary material available at 10.1007/s12553-023-00763-1.","PeriodicalId":12941,"journal":{"name":"Health and Technology","volume":" ","pages":"1-14"},"PeriodicalIF":3.1000,"publicationDate":"2023-06-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10262137/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Health and Technology","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1007/s12553-023-00763-1","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"MEDICAL INFORMATICS","Score":null,"Total":0}

引用次数: 0

Abstract

Background: For the adoption of machine learning clinical decision support systems (ML-CDSS) it is critical to understand the performance aid of the ML-CDSS. However, it is not trivial, how the performance aid should be evaluated. To design reliable performance evaluation study, both the knowledge from the practical framework of experimental study design and the understanding of domain specific design factors are required.

Objective: The aim of this review study was to form a practical framework and identify key design factors for experimental design in evaluating the performance of clinicians with or without the aid of ML-CDSS.

Methods: The study was based on published ML-CDSS performance evaluation studies. We systematically searched articles published between January 2016 and December 2022. From the articles we collected a set of design factors. Only the articles comparing the performance of clinicians with or without the aid of ML-CDSS using experimental study methods were considered.

Results: The identified key design factors for the practical framework of ML-CDSS experimental study design were performance measures, user interface, ground truth data and the selection of samples and participants. In addition, we identified the importance of randomization, crossover design and training and practice rounds. Previous studies had shortcomings in the rationale and documentation of choices regarding the number of participants and the duration of the experiment.

Conclusion: The design factors of ML-CDSS experimental study are interdependent and all factors must be considered in individual choices.

Supplementary information: The online version contains supplementary material available at 10.1007/s12553-023-00763-1.

Abstract Image

查看原文本刊更多论文

在有或没有机器学习临床决策支持系统的帮助下，对临床医生的绩效评估进行系统审查。

背景：对于机器学习临床决策支持系统（ML-CDSS）的采用，了解ML-CDSS的性能辅助至关重要。然而，如何评估性能辅助并非易事。为了设计可靠的性能评估研究，需要从实验研究设计的实践框架中获得知识，并了解特定领域的设计因素。目的：本综述研究的目的是形成一个实用的框架，并确定实验设计的关键设计因素，以评估临床医生在使用或不使用ML-CDSS的情况下的表现。方法：本研究基于已发表的ML-CDSS表现评估研究。我们系统地搜索了2016年1月至2022年12月期间发表的文章。从文章中我们收集了一组设计因素。只考虑了使用实验研究方法比较有或没有ML-CDSS帮助的临床医生的表现的文章。结果：ML-CDSS实验研究设计实用框架的关键设计因素是性能指标、用户界面、地面实况数据以及样本和参与者的选择。此外，我们还确定了随机化、交叉设计、训练和练习轮次的重要性。先前的研究在参与者人数和实验持续时间的选择的基本原理和文件方面存在缺陷。结论：ML-CDSS实验研究的设计因素是相互依存的，在个体选择时必须考虑所有因素。补充信息：在线版本包含补充材料，可访问10.1007/s12553-023-00763-1。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Health and Technology MEDICAL INFORMATICS-

CiteScore

7.10

自引率

0.00%

发文量

期刊介绍： Health and Technology is the first truly cross-disciplinary journal on issues related to health technologies addressing all professions relating to health, care and health technology.The journal constitutes an information platform connecting medical technology and informatics with the needs of care, health care professionals and patients. Thus, medical physicists and biomedical/clinical engineers are encouraged to write articles not only for their colleagues, but directed to all other groups of readers as well, and vice versa.By its nature, the journal presents and discusses hot subjects including but not limited to patient safety, patient empowerment, disease surveillance and management, e-health and issues concerning data security, privacy, reliability and management, data mining and knowledge exchange as well as health prevention. The journal also addresses the medical, financial, social, educational and safety aspects of health technologies as well as health technology assessment and management, including issues such security, efficacy, cost in comparison to the benefit, as well as social, legal and ethical implications.This journal is a communicative source for the health work force (physicians, nurses, medical physicists, clinical engineers, biomedical engineers, hospital engineers, etc.), the ministries of health, hospital management, self-employed doctors, health care providers and regulatory agencies, the medical technology industry, patients'' associations, universities (biomedical and clinical engineering, medical physics, medical informatics, biology, medicine and public health as well as health economics programs), research institutes and professional, scientific and technical organizations.Health and Technology is jointly published by Springer and the IUPESM (International Union for Physical and Engineering Sciences in Medicine) in cooperation with the World Health Organization.