为什么临床试验不能确保人工智能的安全。

IF 5.7 3区 医学 Q1 HEALTH CARE SCIENCES & SERVICES
David P W Rastall, Mohamed Rehman
{"title":"为什么临床试验不能确保人工智能的安全。","authors":"David P W Rastall, Mohamed Rehman","doi":"10.1007/s10916-025-02231-x","DOIUrl":null,"url":null,"abstract":"<p><p>Recent reports have raised concerns about emergent behaviors in next-generation artificial intelligence (AI) models. These systems have been documented selectively adapting their behaviors during testing to falsify experimental outcomes and bypass regulatory oversight. This phenomenon-alignment faking-represents a fundamental challenge to medical AI safety. Regulatory strategies have largely adapted established protocols like clinical trials and medical device approval frameworks, but for next-generation AI these approaches may fail. This paper introduces alignment faking to a medical audience and critically evaluates how current regulatory tools are inadequate for advanced AI systems. We propose continuous logging through \"AI SOAP notes\" as a first step toward transparent and accountable AI functionality in clinical settings.</p>","PeriodicalId":16338,"journal":{"name":"Journal of Medical Systems","volume":"49 1","pages":"98"},"PeriodicalIF":5.7000,"publicationDate":"2025-07-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Why Clinical Trials Will Fail to Ensure Safe AI.\",\"authors\":\"David P W Rastall, Mohamed Rehman\",\"doi\":\"10.1007/s10916-025-02231-x\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<p><p>Recent reports have raised concerns about emergent behaviors in next-generation artificial intelligence (AI) models. These systems have been documented selectively adapting their behaviors during testing to falsify experimental outcomes and bypass regulatory oversight. This phenomenon-alignment faking-represents a fundamental challenge to medical AI safety. Regulatory strategies have largely adapted established protocols like clinical trials and medical device approval frameworks, but for next-generation AI these approaches may fail. This paper introduces alignment faking to a medical audience and critically evaluates how current regulatory tools are inadequate for advanced AI systems. We propose continuous logging through \\\"AI SOAP notes\\\" as a first step toward transparent and accountable AI functionality in clinical settings.</p>\",\"PeriodicalId\":16338,\"journal\":{\"name\":\"Journal of Medical Systems\",\"volume\":\"49 1\",\"pages\":\"98\"},\"PeriodicalIF\":5.7000,\"publicationDate\":\"2025-07-17\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Journal of Medical Systems\",\"FirstCategoryId\":\"3\",\"ListUrlMain\":\"https://doi.org/10.1007/s10916-025-02231-x\",\"RegionNum\":3,\"RegionCategory\":\"医学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q1\",\"JCRName\":\"HEALTH CARE SCIENCES & SERVICES\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Medical Systems","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1007/s10916-025-02231-x","RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"HEALTH CARE SCIENCES & SERVICES","Score":null,"Total":0}
引用次数: 0

摘要

最近的报告引起了人们对下一代人工智能(AI)模型中的紧急行为的担忧。这些系统在测试过程中选择性地调整其行为,以伪造实验结果并绕过监管监督。这种现象——对准伪造——对医疗人工智能安全构成了根本性挑战。监管策略在很大程度上适应了临床试验和医疗设备审批框架等既定协议,但对于下一代人工智能,这些方法可能会失败。本文向医疗受众介绍了对齐,并批判性地评估了当前的监管工具如何不足以适应先进的人工智能系统。我们建议通过“AI SOAP笔记”进行连续记录,作为在临床环境中实现透明和负责任的AI功能的第一步。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
Why Clinical Trials Will Fail to Ensure Safe AI.

Recent reports have raised concerns about emergent behaviors in next-generation artificial intelligence (AI) models. These systems have been documented selectively adapting their behaviors during testing to falsify experimental outcomes and bypass regulatory oversight. This phenomenon-alignment faking-represents a fundamental challenge to medical AI safety. Regulatory strategies have largely adapted established protocols like clinical trials and medical device approval frameworks, but for next-generation AI these approaches may fail. This paper introduces alignment faking to a medical audience and critically evaluates how current regulatory tools are inadequate for advanced AI systems. We propose continuous logging through "AI SOAP notes" as a first step toward transparent and accountable AI functionality in clinical settings.

求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
Journal of Medical Systems
Journal of Medical Systems 医学-卫生保健
CiteScore
11.60
自引率
1.90%
发文量
83
审稿时长
4.8 months
期刊介绍: Journal of Medical Systems provides a forum for the presentation and discussion of the increasingly extensive applications of new systems techniques and methods in hospital clinic and physician''s office administration; pathology radiology and pharmaceutical delivery systems; medical records storage and retrieval; and ancillary patient-support systems. The journal publishes informative articles essays and studies across the entire scale of medical systems from large hospital programs to novel small-scale medical services. Education is an integral part of this amalgamation of sciences and selected articles are published in this area. Since existing medical systems are constantly being modified to fit particular circumstances and to solve specific problems the journal includes a special section devoted to status reports on current installations.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:604180095
Book学术官方微信