Joshua M Biro, Jessica L Handley, James Mickler, Sahithi Reddy, Varsha Kottamasu, Raj M Ratwani, Nathan K Cobb
{"title":"模拟测试对环境数字记录仪评价的价值：一个案例报告。","authors":"Joshua M Biro, Jessica L Handley, James Mickler, Sahithi Reddy, Varsha Kottamasu, Raj M Ratwani, Nathan K Cobb","doi":"10.1093/jamia/ocaf052","DOIUrl":null,"url":null,"abstract":"<p><strong>Objectives: </strong>The objective of this work is to demonstrate the value of simulation testing for rapidly evaluating artificial intelligence (AI) products.</p><p><strong>Materials and methods: </strong>Researcher-physician teams simulated the use of 2 Ambient Digital Scribe (ADS) products by reading scripts of outpatient encounters while using both products, yielding a total of 44 draft notes. Time to edit, perceived amount of effort and editing, and errors in the AI-generated draft notes were analyzed.</p><p><strong>Results: </strong>Ambient Digital Scribe Product A draft notes took significantly longer to edit, had fewer omissions, and more additions and irrelevant or misplaced text errors than ADS Product B. Ambient Digital Scribe Product A was rated as performing better for most encounters.</p><p><strong>Discussion: </strong>Artificial intelligence-enabled products are being rapidly developed and implemented into practice, outpacing safety concerns. Simulation testing can efficiently identify safety issues.</p><p><strong>Conclusion: </strong>Simulation testing is a crucial first step to take when evaluating AI-enabled technologies.</p>","PeriodicalId":50016,"journal":{"name":"Journal of the American Medical Informatics Association","volume":" ","pages":""},"PeriodicalIF":4.7000,"publicationDate":"2025-03-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"The value of simulation testing for the evaluation of ambient digital scribes: a case report.\",\"authors\":\"Joshua M Biro, Jessica L Handley, James Mickler, Sahithi Reddy, Varsha Kottamasu, Raj M Ratwani, Nathan K Cobb\",\"doi\":\"10.1093/jamia/ocaf052\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<p><strong>Objectives: </strong>The objective of this work is to demonstrate the value of simulation testing for rapidly evaluating artificial intelligence (AI) products.</p><p><strong>Materials and methods: </strong>Researcher-physician teams simulated the use of 2 Ambient Digital Scribe (ADS) products by reading scripts of outpatient encounters while using both products, yielding a total of 44 draft notes. Time to edit, perceived amount of effort and editing, and errors in the AI-generated draft notes were analyzed.</p><p><strong>Results: </strong>Ambient Digital Scribe Product A draft notes took significantly longer to edit, had fewer omissions, and more additions and irrelevant or misplaced text errors than ADS Product B. Ambient Digital Scribe Product A was rated as performing better for most encounters.</p><p><strong>Discussion: </strong>Artificial intelligence-enabled products are being rapidly developed and implemented into practice, outpacing safety concerns. Simulation testing can efficiently identify safety issues.</p><p><strong>Conclusion: </strong>Simulation testing is a crucial first step to take when evaluating AI-enabled technologies.</p>\",\"PeriodicalId\":50016,\"journal\":{\"name\":\"Journal of the American Medical Informatics Association\",\"volume\":\" \",\"pages\":\"\"},\"PeriodicalIF\":4.7000,\"publicationDate\":\"2025-03-21\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Journal of the American Medical Informatics Association\",\"FirstCategoryId\":\"91\",\"ListUrlMain\":\"https://doi.org/10.1093/jamia/ocaf052\",\"RegionNum\":2,\"RegionCategory\":\"医学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q1\",\"JCRName\":\"COMPUTER SCIENCE, INFORMATION SYSTEMS\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of the American Medical Informatics Association","FirstCategoryId":"91","ListUrlMain":"https://doi.org/10.1093/jamia/ocaf052","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"COMPUTER SCIENCE, INFORMATION SYSTEMS","Score":null,"Total":0}
引用次数: 0
摘要
目的：这项工作的目的是证明模拟测试对快速评估人工智能（AI）产品的价值。材料和方法：研究人员-医生团队模拟了两种Ambient Digital Scribe （ADS）产品的使用，在使用这两种产品的同时，通过阅读门诊病人的病历，得出了总共44份草稿笔记。分析了编辑时间、感知到的工作量和编辑量以及人工智能生成的草稿笔记中的错误。结果：与ADS产品b相比，Ambient Digital Scribe产品A的草稿笔记编辑时间明显更长，遗漏更少，增加的内容和不相关或放错位置的文本错误更多。讨论：人工智能支持的产品正在迅速开发并付诸实践，超越了安全问题。仿真测试可以有效地识别安全问题。结论：在评估人工智能技术时，模拟测试是关键的第一步。
The value of simulation testing for the evaluation of ambient digital scribes: a case report.
Objectives: The objective of this work is to demonstrate the value of simulation testing for rapidly evaluating artificial intelligence (AI) products.
Materials and methods: Researcher-physician teams simulated the use of 2 Ambient Digital Scribe (ADS) products by reading scripts of outpatient encounters while using both products, yielding a total of 44 draft notes. Time to edit, perceived amount of effort and editing, and errors in the AI-generated draft notes were analyzed.
Results: Ambient Digital Scribe Product A draft notes took significantly longer to edit, had fewer omissions, and more additions and irrelevant or misplaced text errors than ADS Product B. Ambient Digital Scribe Product A was rated as performing better for most encounters.
Discussion: Artificial intelligence-enabled products are being rapidly developed and implemented into practice, outpacing safety concerns. Simulation testing can efficiently identify safety issues.
Conclusion: Simulation testing is a crucial first step to take when evaluating AI-enabled technologies.
期刊介绍：
JAMIA is AMIA''s premier peer-reviewed journal for biomedical and health informatics. Covering the full spectrum of activities in the field, JAMIA includes informatics articles in the areas of clinical care, clinical research, translational science, implementation science, imaging, education, consumer health, public health, and policy. JAMIA''s articles describe innovative informatics research and systems that help to advance biomedical science and to promote health. Case reports, perspectives and reviews also help readers stay connected with the most important informatics developments in implementation, policy and education.
求助内容：
应助结果提醒方式：
京公网安备 11010802042870号
