机器学习预警模型在内科和外科住院病人中的可信度。

IF 3.4 Q2 HEALTH CARE SCIENCES & SERVICES
JAMIA Open Pub Date : 2025-01-06 eCollection Date: 2025-02-01 DOI:10.1093/jamiaopen/ooae156
Pedro J Caraballo, Anne M Meehan, Karen M Fischer, Parvez Rahman, Gyorgy J Simon, Genevieve B Melton, Hojjat Salehinejad, Bijan J Borah
{"title":"机器学习预警模型在内科和外科住院病人中的可信度。","authors":"Pedro J Caraballo, Anne M Meehan, Karen M Fischer, Parvez Rahman, Gyorgy J Simon, Genevieve B Melton, Hojjat Salehinejad, Bijan J Borah","doi":"10.1093/jamiaopen/ooae156","DOIUrl":null,"url":null,"abstract":"<p><strong>Objectives: </strong>In the general hospital wards, machine learning (ML)-based early warning systems (EWSs) can identify patients at risk of deterioration to facilitate rescue interventions. We assess subpopulation performance of a ML-based EWS on medical and surgical adult patients admitted to general hospital wards.</p><p><strong>Materials and methods: </strong>We assessed the scores of an EWS integrated into the electronic health record and calculated every 15 minutes to predict a composite adverse event (AE): all-cause mortality, transfer to intensive care, cardiac arrest, or rapid response team evaluation. The distributions of the First Score 3 hours after admission, the Highest Score at any time during the hospitalization, and the Last Score just before an AE or dismissal without an AE were calculated. The Last Score was used to calculate the area under the receiver operating characteristic curve (ROC-AUC) and the precision-recall curve (PRC-AUC).</p><p><strong>Results: </strong>From August 23, 2021 to March 31, 2022, 35 937 medical admissions had 2173 (6.05%) AE compared to 25 214 surgical admissions with 4984 (19.77%) AE. Medical and surgical admissions had significant different (<i>P</i> <.001) distributions of the First Score, Highest Score, and Last Score among those with an AE and without an AE. The model performed better in the medical group when compared to the surgical group, ROC-AUC 0.869 versus 0.677, and RPC-AUC 0.988 versus 0.878, respectively.</p><p><strong>Discussion: </strong>Heterogeneity of medical and surgical patients can significantly impact the performance of a ML-based EWS, changing the model validity and clinical discernment.</p><p><strong>Conclusions: </strong>Characterization of the target patient subpopulations has clinical implications and should be considered when developing models to be used in general hospital wards.</p>","PeriodicalId":36278,"journal":{"name":"JAMIA Open","volume":"8 1","pages":"ooae156"},"PeriodicalIF":3.4000,"publicationDate":"2025-01-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11702360/pdf/","citationCount":"0","resultStr":"{\"title\":\"Trustworthiness of a machine learning early warning model in medical and surgical inpatients.\",\"authors\":\"Pedro J Caraballo, Anne M Meehan, Karen M Fischer, Parvez Rahman, Gyorgy J Simon, Genevieve B Melton, Hojjat Salehinejad, Bijan J Borah\",\"doi\":\"10.1093/jamiaopen/ooae156\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<p><strong>Objectives: </strong>In the general hospital wards, machine learning (ML)-based early warning systems (EWSs) can identify patients at risk of deterioration to facilitate rescue interventions. We assess subpopulation performance of a ML-based EWS on medical and surgical adult patients admitted to general hospital wards.</p><p><strong>Materials and methods: </strong>We assessed the scores of an EWS integrated into the electronic health record and calculated every 15 minutes to predict a composite adverse event (AE): all-cause mortality, transfer to intensive care, cardiac arrest, or rapid response team evaluation. The distributions of the First Score 3 hours after admission, the Highest Score at any time during the hospitalization, and the Last Score just before an AE or dismissal without an AE were calculated. The Last Score was used to calculate the area under the receiver operating characteristic curve (ROC-AUC) and the precision-recall curve (PRC-AUC).</p><p><strong>Results: </strong>From August 23, 2021 to March 31, 2022, 35 937 medical admissions had 2173 (6.05%) AE compared to 25 214 surgical admissions with 4984 (19.77%) AE. Medical and surgical admissions had significant different (<i>P</i> <.001) distributions of the First Score, Highest Score, and Last Score among those with an AE and without an AE. The model performed better in the medical group when compared to the surgical group, ROC-AUC 0.869 versus 0.677, and RPC-AUC 0.988 versus 0.878, respectively.</p><p><strong>Discussion: </strong>Heterogeneity of medical and surgical patients can significantly impact the performance of a ML-based EWS, changing the model validity and clinical discernment.</p><p><strong>Conclusions: </strong>Characterization of the target patient subpopulations has clinical implications and should be considered when developing models to be used in general hospital wards.</p>\",\"PeriodicalId\":36278,\"journal\":{\"name\":\"JAMIA Open\",\"volume\":\"8 1\",\"pages\":\"ooae156\"},\"PeriodicalIF\":3.4000,\"publicationDate\":\"2025-01-06\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11702360/pdf/\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"JAMIA Open\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/10.1093/jamiaopen/ooae156\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"2025/2/1 0:00:00\",\"PubModel\":\"eCollection\",\"JCR\":\"Q2\",\"JCRName\":\"HEALTH CARE SCIENCES & SERVICES\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"JAMIA Open","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1093/jamiaopen/ooae156","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2025/2/1 0:00:00","PubModel":"eCollection","JCR":"Q2","JCRName":"HEALTH CARE SCIENCES & SERVICES","Score":null,"Total":0}
引用次数: 0

摘要

目的:在综合医院病房,基于机器学习(ML)的早期预警系统(ews)可以识别有恶化风险的患者,以促进抢救干预。我们评估了基于ml的EWS对普通医院病房住院的内科和外科成年患者的亚群表现。材料和方法:我们评估了整合到电子健康记录中的EWS评分,并每15分钟计算一次,以预测复合不良事件(AE):全因死亡率、转入重症监护、心脏骤停或快速反应小组评估。计算入院后3小时的第一次评分、住院期间任何时间的最高分、发生AE或无AE出院前的最后一次评分的分布。最后得分用于计算受试者工作特征曲线(ROC-AUC)和精密度-召回率曲线(PRC-AUC)下的面积。结果:2021年8月23日至2022年3月31日,内科住院35 937例发生2173例(6.05%)AE,外科住院25 214例发生4984例(19.77%)AE。讨论:内科和外科患者的异质性会显著影响基于ml的EWS的表现,改变模型的效度和临床识别。结论:目标患者亚群的特征具有临床意义,在开发用于综合医院病房的模型时应考虑到这一点。
本文章由计算机程序翻译,如有差异,请以英文原文为准。

Trustworthiness of a machine learning early warning model in medical and surgical inpatients.

Trustworthiness of a machine learning early warning model in medical and surgical inpatients.

Trustworthiness of a machine learning early warning model in medical and surgical inpatients.

Trustworthiness of a machine learning early warning model in medical and surgical inpatients.

Objectives: In the general hospital wards, machine learning (ML)-based early warning systems (EWSs) can identify patients at risk of deterioration to facilitate rescue interventions. We assess subpopulation performance of a ML-based EWS on medical and surgical adult patients admitted to general hospital wards.

Materials and methods: We assessed the scores of an EWS integrated into the electronic health record and calculated every 15 minutes to predict a composite adverse event (AE): all-cause mortality, transfer to intensive care, cardiac arrest, or rapid response team evaluation. The distributions of the First Score 3 hours after admission, the Highest Score at any time during the hospitalization, and the Last Score just before an AE or dismissal without an AE were calculated. The Last Score was used to calculate the area under the receiver operating characteristic curve (ROC-AUC) and the precision-recall curve (PRC-AUC).

Results: From August 23, 2021 to March 31, 2022, 35 937 medical admissions had 2173 (6.05%) AE compared to 25 214 surgical admissions with 4984 (19.77%) AE. Medical and surgical admissions had significant different (P <.001) distributions of the First Score, Highest Score, and Last Score among those with an AE and without an AE. The model performed better in the medical group when compared to the surgical group, ROC-AUC 0.869 versus 0.677, and RPC-AUC 0.988 versus 0.878, respectively.

Discussion: Heterogeneity of medical and surgical patients can significantly impact the performance of a ML-based EWS, changing the model validity and clinical discernment.

Conclusions: Characterization of the target patient subpopulations has clinical implications and should be considered when developing models to be used in general hospital wards.

求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
JAMIA Open
JAMIA Open Medicine-Health Informatics
CiteScore
4.10
自引率
4.80%
发文量
102
审稿时长
16 weeks
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:604180095
Book学术官方微信